CC BY
41
A study to Relate the Quality of Forum Messages with the FLOW Experience
Steven Abrantes, Luis B. Gouveia., and Yousef Ibrahim Daradkeh
Abstract—The use of online discussion forum as a tool to enhance learning and student communication has been widely discussed in literature. Despite all the potential enclosed on such tool, most of the times, little or no concern is made regarding the messages content quality. This paper presents a study that relate the message quality of an online discussion forum with the concept of flow experience, in order to provide a strategy to improve the value of this tool to support learning and the learning experience among a group of students. Based on the presented study, we can say that students that have an average value of the flow experience are those who participate more in a collaborative environment.
Keywords—Forum messages, quality, flow experience, online discussion forum evaluation.
I. INTRODUCTION
Information and communication technologies have already been integrated in our current education systems. Some teachers have adopted those technologies in classroom context, modifying the traditional education system, based on a board, chalk and a set of slides. However, there are still teachers who tend to resist to the new information and communication innovations.
Despite the potential that the information and communication technologies brought to our today's education, the use of these in schools have been shown as incoherent and in many cases, ineffective [14]. One reason for this is the challenge for teachers to integrate technology into their classrooms. The use of technology in the classroom requires both pedagogical and technical knowledge and therefore a substantial investment of time and resources, both for the institution and teacher [14].
The adoption of technologies for teaching and learning is an innovation that challenges the structure, culture and practice of universities and higher education institutions [1].
The introduction of the information and communication technologies, by some teachers, in a given environment, has a long tradition of being based in knowledge transmission throughout a classroom, which can be seen as a classic case of a diffusion of innovation [1].
Due to the increased use of information and communication in the context of higher education, we can
Steven Abrantes is with Institute Polytechnic of Viseu, e-mail: steven@di.estv.ipv.pt).
Luis. B. Gouveia is with University Fernando Pessoa, e-mail: dlmbg@ufp.edu.pt)
Yousef Ibrahim Daradkeh is with Prince Sattam bin Abdulaziz University, e-mail: daradkehy@yahoo.ca)
see a growing use of online discussion forums by those involved in education [10]. Also, more recently, a number of Web 2.0 tools are in place. However, the use of online discussion forums can provide a number of advantages for the teacher
Also, online discussion forums have the advantage of leaving all that was discussed recorded to then be analyzed and discussed later [10], allowing the realization of studies like the one presented here.
The problem of evaluation, associated with the use of online discussion forums, has been a relevant aspect when instilled in the process of evaluating a particular course. Evaluation may be considered a very complex process leading to several questions and uncertainties for the evaluators. Another aspect related with the interaction of the users with games has to see with the flow experience introduced by [3].
The experience of the flow means the sensation that people feel when they are completely involved in what they are doing, that is, people like the experience and want repeat it [4]. This means that for students to be involved with games, it is necessary that they presence the flow state. The theory of the flow allows us to measure the interaction of users with the computer systems, verifying if these are more or less playfulness [16].
II. A TING ONLINE DISCUSSION FORUMS
Although the use of forums in the context of higher education is already widely used, some issues associated with its utilization arise, such as, what is its potential and how can we make its own evaluation. The evaluation issue is quite complex and raises many questions and uncertainties to the evaluator. According to Santos [6], this fact "... certainly has to do with the meanings and concepts of assessment practices that each teacher has, as well as their own evaluative experience" [15]. So what does the term "evaluate" mean? In the dictionary [13], the term "evaluate" means "to determine the value of', "understand", "judge", "appreciate". Evaluating student's results is an understanding, appreciation and judgment of their work, by the teacher, using different set of instruments in order to determine a qualitative or quantitative value.
With the simple counting of posts of each participant in an online discussion forum, you cannot measure the quality of interactions. Moreover, we can state that quality is not synonymous with quantity [6].
Meyer used four different kinds of methods to analyze seventeen online forums of a doctoral program in order to
validate its efficiency [10]. In particular, for the present study, we considered the approach proposed by [9], who follows a model that basically follows three steps:
Classify each message of each student as being significant or not significant. This is, messages like "Thank you", "until tomorrow", "Hello", are classified as non-significant and other messages that are related to the content of the topic in question are classified as significant.
Once each message has been classified, we should classify each one according to a scale of 1 to 3 (1 - Positive, 2 -Good, 3 - Very Good). Finally, calculate the number of meaningful messages through their multiplication factor, this is, multiply the number of messages with a classification of very good by three, multiply the messages with a classification of good by two and finally multiply the messages with a classification of positive by 1, adding in the end, all these components. After this operation is performed, it is necessary to convert these values to a qualitative classification. As for the conversion of these values we can use as basis, the student who has more meaningful messages, this will be awarded with 20 points and the others will use the direct proportionality. In this model, the student who has written more posts does not necessarily have better ratings than the student who has participated less. This is the algorithm described by Mesquita [9], that serves as the base for the current evaluation of the quality and the participation of the students in an online discussion forum. This approach assumes that we are in a collaborative learning environment and that the teacher has with him an evaluation grid in order to grade each of the messages of the various participants.
In conclusion, the formula follows:
Partial classification of the student = nrespx * ntipol + nrespx * ntipo2 + nrespx * ntipo3.
Where nrespx represents the number of significant responses and ntipo refers to a scale of 1 to 3 (1 - Positive, 2 - Good, 3 - Very Good). The student's final grade is calculated on the basis of the student who has more meaningful messages (partial classification of the student) who will be awarded with 20 points and the other using the proportionality rule.
III. The flow experience
An aspect related with the interaction of the users with collaborative environments has to see with the flow experience introduced by [3]. The experience of the flow means the sensation that people feel when they are completely involved in what they are doing, that is, people like the experience and want repeat it [4]. This means that for students to be involved with collaborative environments, it is necessary that they presence the flow state.
The theory of the flow allows us to measure the interaction of users with the computer systems, verifying if these are more or less playfulness [16]. The flow experience is used in this article to characterize the interaction between
the human and the new technologies [16].
When one is in the presence of the flow experience, this will bring to the users, a sense of pleasure of what he is doing. This satisfaction will encourage the user to repeat the task again [17].
Csikszentmihalyi says that a person who is in the presence of the flow state has the following characteristics [3], [5]:
Clear goals and immediate feedback;
Equilibrium between the level of challenge and personal skill;
• Merging of action and awareness;
• Focused concentration;
• Sense of potential control;
• Loss of self-consciousness;
• Time distortion;
• Autotelic or self-rewarding experience.
For a person to be in the presence of the flow experience it is necessary a balance between the level of challenge and personal skill [4], (Ошибка! Источник ссылки не найден.).
Skills
Figure 1. Flow Experience (Csikszentmihalyi, 1982).
The sensation of an excellent experience in the accomplishment of any daily task is our reason of living. If we do not feel this excellent experience with our everyday tasks, we will question our self, if it is worth living [4].
Previous researches have used the flow experience to measure playfulness, involvement, satisfaction and other states with the involvement in computational environments [2], [7], [11], [12], [16].
Trevino and Webster (1992) define four dimensions for the flow experience:
• Control;
• Attention Focus;
• Curiosity;
• Intrinsic Interest.
There is one more dimension, sense of time, that is also important to measure the flow state [8].
Control
Individuals should experience, feelings in control, within computer interactions [3].
Attention Focus
Attention focus is another important element of flow. When individuals are in the flow state, their minds are narrowed to what they are doing, filtering out irrelevant thoughts and perceptions [17].
Curiosity
Curiosity is aroused when in the flow state. The curiosity sensation can be aroused through varied, new and admirable stimulations. For example, the new technologies will be able to cause this sensation of curiosity through colors and sounds [17].
Intrinsic Interest
When people feel they are in the flow state, these are involved for the amusement and pleasure [17].
Sense of time
When people feel they are in the flow state, there is a perceptual transformation of time, characterized by the sensation of time slowing down or speeding up [8].
People who interact with computers, with an entertainment spirit, transmit a much more positive experience, of those, who are in the computer for obligation [17].
IV. The study
This experiment was carried through, involving students from a university school. The main tool used was Google Groups, for this experiment. This section presents the carried through experiment, the data obtained, as well as the statistical procedures applied.
Previously to this study, a test with five students was done, to analyze the effectiveness of the survey. From this previous study, we concluded that some questions were ambiguous for the population studied.
The survey was passed through the Internet with the help of "Lime Survey". The data collection was performed in the first week of November 2009.
The Instruments used were Google Groups, Google Docs and Face book and a survey consisting on some questions, in order to classify the students in terms of innovation and also to measure the type of messages that these students send to a discussion forum.
4.1 SAMPLE
This study intends to determine if the students inquired are in the flow state and the quality of the messages that these have sent to the discussion forum. The data has been collected through one hundred and twelve surveys of students. The surveys have been submitted to a rigorous test, having not excluded any individual; therefore, the sample
consisted on one hundred and twelve valid surveys. The criteria of exclusion of inquiries were: students who had not discriminated their sex or age in the survey; students with incoherent answers throughout the survey (e.g answers that always presented values in the extremities of the scales, or incompatible); students who left 80% of the survey in blank. Once, one hundred and twelve valid inquiries were obtained, the sample is considered sufficiently satisfactory.
4.2 DATA Analysis In order to classify the category of the respondents belonging to the initial market (innovators, early adopters) and the majority market (early majority, late majority and laggards), the scores of individual innovation developed by Anderson, Varnhagen and Campbell (1999) was used. This scoring process was developed based on the assumption that users of the initial market used the technology sooner and gained more experience when compared with the majority market [1]. We used a scale (6 - none to 1 - Intensively) for each type of applications used (Google Docs, Google Groups and Face book), before and after the completion of the project. The result is the sum of the six responses. The minimum value of total responses was 6, which would classify the most innovative. The maximum total number of answers would be 36, which would be the classification of the least innovative. The values of innovation were between16 and 31.
For the cumulative frequencies, we found that first 16%, i.e., the initial market contains 14 respondents. The next 84%, which are those that belong to the market majority, consists on 67 respondents. Those who belong to the latter group are those with the highest values, which mean they are less innovative than those belonging to the first 16% of the graph of cumulative frequencies.
4.2.1 Quality of the messages and the Flow
experience
Relatively to the evaluation of the students for online discussion forums, we can concluded that there has been a total of 661 messages, where 238 where messages that has been classified as Very Good, 150 as Good, 203 as Positive and 70 of the messages has been classified as not significant, this is, these messages were considered not being valid for the discussion between the participants. Separating these messages for the students who have used do laptop and the desktop, we can reach to the conclusion that the students who have used the laptop have sent more messages (455) then the students who have used the desktop (136).
For the users who used the laptop, 185 were considered Very Good, 113 were Good, 157 classified as Positive and 45 classified as not significant. As for the users of the desktop, 53 were messages classified as Very Good, 37 classified as Good, 46 as Positive and 25 as not significant. However, we need to consider the fact that the number of users using the laptop is greater than the number of the desktop users. As result, we provide in table 1 the average number of messages sent by each student for the laptop and desktop in order to allow a comparison based on relative numbers and taking into account the different dimension of
the two groups.
Table 1: Number of messages.
Number of messages
Significant Not Significant Total (Significant)
3 2 1
238 150 203 70 1217
Table 2: Average number of messages.
Average number of messages
N° TWO1 MMW2 TW3 MMW4
112 591 5.27 1217 10.86
1 - Total without multiplication factor
2 - Average messages without multiplication factor
3 - Total with multiplication factor
4 - Average messages with multiplication factor
As we can conclude from Table 1, the students have sent and average of 5.27 message to the discussion forum.
In order to determine the presence of the flow experience for each type of device, it was verified that, on average, the students were above value three (Likert scale of five points), that is, the majority of the students, are in the presence of the flow experience, for the five variables mentioned for this study (attention focus, curiosity, control, intrinsic interest and sense of time).
If we cross the information of the quality of the messages with the flow experience, we can see that students that have a medium value for the flow experience, were those who sent more messages.
Table 3: Medium number of messages/ Flow Experience.
Medium Messages Flow Exp. Scale N° Students
0 0 a 1 1
0.035 1 a 2 1
10.669 2 a 3 18
8.535 3 a 4 86
0.714 4 a 5 7
V. CONCLUSION
In order to evaluate the use of collaborative environments, it was performed an experiment involving students of higher education. This study has the main objective to validate if students are in the flow experience and also measure the quality of the messages that students sent to a collaborative environment.
In order to determine the presence of the flow experience, it was verified that, on average, the students were above the medium value of the flow experience (Likert scale of five points), this is, the majority of the students, are in the presence of the flow experience, for the five variables mentioned for this study (attention focus, curiosity, control,
intrinsic interest and sense of time).
This report also proposed a formula that allows us to measure the quality of the interventions by the various participants in an online discussion forum. It can be considered, that this algorithm is one of the possible ways, among others, to assess the participation of online discussion forums.
To use this algorithm to evaluate a online discussion forum it is necessary that the evaluator has the following basic elements: an online discussion forum, a group of students that interact on the forum, a unique identifier for each participant, a set of messages sent by each of the participants and an evaluation grid, as described above, so that the evaluator can mark each intervention for each participant. The analysis of data allows us to conclude that the students sent a total of 591 messages, being 238 were classified as Very Good, 150 Good, 203 classified as Positive and 70 classified as not significant. Considering the average number of messages, each user sent 5.27 messages. Another purpose of this study is to classify the type of messages sent by each of the different kind of states of the flow experience (Very Good, Good, Positive and not significant).
Regarding the number of messages sent by each kind of students in terms of the flow experience, we concluded that the students that have a medium value of the flow experience (2 a 3), are those who send more messages. With these statements we can say that students that have an average value of the flow experience are those who participate more in a collaborative environment.
REFERENCES
[1] ANDERSON, T., VARNHAGEN, S. & CAMPBELL, K. 1998. Faculty adoption of teaching and learning technologies: Contrasting earlier adopters and mainstream faculty. Canadian Journal of Higher Education, 28, 71-98.
[2] CHEN, H., WIGAND, R. T. & NILAN, M. 2000. Exploring Web users' optimal flow experiences. Information Technology & People, 12.
[3] CSIKSZENTMIHALYI, M. 1975. Beyond Boredom and anxiety, San Francisco, CA.
[4] CSIKSZENTMIHALYI, M. towards a Psychology of Optimal Experience. Annual Review of Personality and Social Psychology, 1982.
[5] CSIKSZENTMIHALYI, M. 1990. The psychology of optimal experience, Harper Collins.
[6] DROPS, G. 2003. Assessing Online Chat Sessions. Online Cl@ssroom, 1-8.
[7] GHANI, J. & DESHPANDE, S. 1994. Task Characteristics and the Experience of Optimal Flow in Human-Computer Interaction. The Journal of Psychology, 128, 381-391.
[8] MCKENNA, K. & LEE, S. 2005. A Love Affair with MUDs: Flow and Social Interaction in Multi-UserDungeons [Online]. Available: http://fragment.nl/mirror/various/McKenna_et_al.nd.A_love_affair_w ith_muds.html [Accessed 2007/10/10/.
[9] MESQUITA, J. S. 2007. b-learning no ensino secundario recorrente -Uma Proposta baseada na construjao do conhecimento. Tese de Mestrado, Universidade de Aveiro.
[10] MEYER, K. A. 2004. Evaluating online discussions: Four different frames of analysis. Journal of Asynchronous Learning Networks, 8, 101-114.
[11] NOVAK, T. P. & HOFFMAN, D. L. 1997. Measuring the Flow Experience Among Web Users. Vanderbilt University.
[12] NOVAK, T. P., HOFFMAN, D. L. & YUNG, Y. 2000. Measuring the Customer Experience in Online Environments: A Structural Modeling Approach. Marketing Science, 19, 22-42.
[13] PRIBERAM. 2009. Dicionário Electrónico de Língua Portuguesa [Online]. Available: http://www.priberam.pt/dlpo/default.aspx7paHavaliar [Accessed 812-2009.
[14] REINDERS, H. 2009. Teaching (with) technology. The scope and practice of teacher education for technology. Prospect. Prospect Journal, 24, 15-23.
[15] SANTOS, A. 2000. Ensino à distância & Tecnologias de informaçâo, FCA.
[16] TREVINO, L. K. & WEBSTER, J. 1992. Flow in computer-mediated communication. Communication Research, 19, 539-573.
[17] WEBSTER, J., TREVINO, L. K. & RYAN, L. 1993. The dimensionality and correlates of flow in human-computer interaction. Computer Game Research, 9, 411-426.
Анализ качества сообщений на форуме с
помощью потоков
Стевен Абрантес, Луис Гувейя, Юозеф Дарадкех
Аннотация— Использование интернет-форумов (обсуждений) в качестве инструмента для повышения качества обучения и связи со студентами широко обсуждается в литературе. Несмотря на все возможности таких инструментов, в большинстве случаев, мало внимания уделяется качеству (содержанию) сообщения. Эта статья представляет собой исследование, в котором оценивается качество сообщений онлайнового дискуссионного форума при помощи концепции потоков.
Ключевые слова—сообщения на форуме, качество, анализ потоков, оценка онлайн-форумов.
