CC BY
2
Azimova Sh.S. senior teacher Tashkent financial institute
COMPARISON OF ONLINE AND OFFLINE QUESTIONNAIRES FOR CONDUCTING PSYCHOLOGICAL RESEARCH IN THE MODERN EDUCATIONAL ENVIRONMENT
Annotation. The uploading, exchanging, and gathering of data for research and data pertaining to psychology can be done effectively on the Internet. A new area of study in the social and behavioral sciences may be opened up by the relative affordability and convenience of building and maintaining Web-based apps, along with the ease of use offered by form-based surveys' graphical user interfaces. This study compared Web-based assessment methods with conventional paper-based methods of various measures of Internet attitudes and behaviors in an Uzbek population in order to investigate the potential use of Internet tools in psychological research. To find variations between the two samples as well as between the psychometric properties of the questionnaires, the collected data were evaluated. Even while we discovered substantial changes in the attitudes and behaviors on the Internet between the two populations, there were no discernible variations in the psychometric characteristics of the various questionnaires. Given the lack of control over the features of the online sample, this result, which is similar to those previously observed in Web-based assessments of personality traits, is even more intriguing. These results show that Internet-based questionnaires may be a suitable substitute for more conventional paper-based measurements if sampling control and validity assessment are included.
Key words: Internet, online, offline, data, psychology.
INTRODUCTION
Many researchers suggest that the Internet can be a useful medium for the gathering and exchange of information in research and data pertaining to psychology. Because of the benefits it can provide to experimental psychology, the Web approach is generating a lot of interest in the scientific community. In a recent study, Musch and Reips spoke with psychologists who had previously participated in Internet studies in an effort to determine the potential benefits of this strategy. The following are the key benefits cited:
• the vast population access, which allows for easier generalization of the results and a higher level of external validity;
• less expensive experimental expenditures and the potential to eliminate all the annoyances associated with using laboratories, such as scheduling, space restrictions, sharing equipment, and so forth;
• the potential to make the tools available 24/7, without any time restrictions;
• involvement that is entirely optional, which typically increases responders' motivation.
The Web may prove superior to paper, as Miller and colleagues recently noted, because it "potentially provides increased accessibility; capability for dynamic and interactive forms, which eliminates the viewing of irrelevant questions and customized feedback tailored to the content of the responses." The ability to access a sizable global pool of participants at any time makes the Internet particularly appealing for psychological research. The Web provides both a vast demographic range for conducting experiments and the potential to reach a specific sample with a particular set of characteristics. Since 1997, an increasing number of authors have made an effort to pinpoint and clarify the methodological steps necessary to create a reliable experimental design.
Using a within-subjects approach, John Krantz and colleagues' online study on the factors influencing female beauty got underway. The first psychology Web study to be published in a scholarly journal was this one. However, there are distinct drawbacks to conducting research online:
• Web users have a variety of hardware, software, and Internet connections, it is challenging to regulate the learning environment. There is no way to guarantee that each person taking part in the experiment will be exposed to the identical sounds, colors, or time cues;
• Study participants are frequently not watched, the researcher cannot be certain of the data gathered. Members of online groups frequently change their gender or create phony "nickname" identities and publicly accept them in others;
• Online experiment participants are self-selected and in no way random samples of the overall population. They tend to be more concentrated towards the upper end of the social and educational spectrum, in particular;
• For a psychologist, developing a Web-based assessment tool is not a simple undertaking. Typically, it involves managing the database where the answers are recorded as well as creating various Web sites.
As Buchanan emphasized, "There are undoubtedly a number of practical concerns that should be taken into account by anyone wishing to use online examinations, as well as significant theoretical issues that should be answered. Online clinical examinations are both desirable and risky, to sum up. There is undoubtedly a lot of potential, but much work remains before this potential can be realized. Whether these instruments will be beneficial in behavioral telehealth scenarios is something that only time and in-depth research will be able to answer. The biggest issue facing Internet researchers right now is how to define fundamental methodological concepts so as to have more control over the internal and statistical validity of experiments.
This study compared Web-based assessment techniques with traditional paper-based methods of different measures of Internet attitude in a sample of
Uzbek students in order to investigate the potential use of Internet tools in psychological research. On one side, we gave a sample of college students a paper-and-pencil questionnaire; on the other, we created an HTML page and an online database and posted the same questionnaire on the Web. We specifically sought to determine:
• any variation in the responses gathered;
• any variations in the evaluation tools' psychometric characteristics.
METHOD AND MATERIALS
All who participated in the digital version
Online questionnaires totaling 233 were finished. 100 participants were male (51.2%), and 99 were female (48.8%). The sample's age ranged from 16 to 41 years, with a mean age of 23.8 (SD = 4.095). All of the participants were found through postings on search engines, newsgroups, and emails.
All who took part in the offline variant
202 undergraduate students who attended a major Uzbek university were recruited for the study using fliers and presentations made during various classes. The sample's age ranged from 21 to 28 years, with an average age of 22.96 (SD = 1.57). 128 men (63.4%) and 74 women (36.6%) made up the sample.
Survey questions
The survey's questionnaire is divided into three sections.
1. The first section concentrated on traditional demographic factors (age, sex, education level) as well as indicators of computer and Internet use.
2. The Computer Use Survey by Pratarelli et al. made up part two. This tool consists of 74 true/false questions covering a wide spectrum of online social and personal activities as well as testing potential problematic habits.
3. The Internet Attitudes Survey by Weiser, which consists of 20 items with Likert scales about personal and professional Internet use, makes up the final section.
Processes
Entire classes received the paper-based survey at once during the day. The experimenter gave each participant a packet of paper-based self-report tests. All of the participants were instructed to complete the questionnaire completely, honestly as they could, and without discussing their answers with anybody nearby. Each time they finished; they dropped their finished packet in a safe drop box that was under the experimenter's control. By visiting a certain Web site, participants finished the online survey. The Ms FrontPage 2000 editor was used to create the HTML for this website, which is hosted on a Microsoft NT server with FrontPage extensions. Participants were encouraged to submit their data after completing the survey.
Participants were encouraged to submit their data after completing the survey. The information was automatically saved in a file with a tab-delimited format after submission and was taken away from participants. Participants were reminded that both versions were entirely anonymous and that their answers were
to be treated as confidential. For the statistical analyses, SPSS 10.1 for Windows was used.
RESULTS
Online usage
The two samples were initially examined with regard to the characteristics related to Internet use. The choices for both samples were generally similar: the most popular Internet tools are e-mail and Web browsing, followed distantly by chats and Usenet, MUDs (Multi User Dungeons), and FTP (File Transfer Protocol). These decisions highlighted a distinct preference for tools catered to information research and interpersonal communication (Tables 1 -3). The data also revealed that our samples do not frequently utilize systems like MUDs or FTP, which have a sizable user base in other nations, like the United States and northern Europe (see Tables 2 and 3). We used chi-square analysis to compare the two samples. Both the use of email (Chi-square = 18.1, d.f. = 4, p 0.001) and web browsing (Chi-square =13. 7, d.f. = 4, p 0.008) showed significant differences. We have a stronger use of the two tools and nearly no subjects have ever used them in the sample we recruited online, in particular. Instead, 14 to 16% of the offline sample have never used an Internet tool before.
TABi.t: 1. TJME; Si'JiN'i EAt H WjlELK HN E-MALIT NEWSGROUP^ IVILB SuiunNG
Subjects (%)
E-mail Nen?i£rc>ttf)£ Web&afing
On- Off tijf- Off. On- Off
Time line litie !h le tine line title
Nk-ver tis& 1 13-9 H2.fi 86.6 2.5 163
1 -2 hrs 42.9 33.7 13 fi 9.4 37.4 26.7
3-5 hrs 25 1 31.2 3 4 2 28.6 35.fi
6-9 hrs 21.7 ] 9.8 2 24.6 18.3
More than 9.4 1.5 6.9 3
10 hrs
TABLE 3. TLM t SrtN r HAC.:H W t:tK i JN CHAT RI * JMS and FTP
Subjects (%)
Chat Rooms
FTP
On- Off On- Off-
Time line line line line
Never use 42.4 36.1 55.5 57.9
]-2hrs 28.6 33.2 36.5 36.6
2-5 hrs 22.7 23.8 6.4 4^5
5-]5hrs 5.9 6.4 0.5
More than 0.5 0.5 1.5 0.5
15hrs
Examination of the psychometric traits
If the factor structures of the Web-based and paper-based surveys were the same, various exploratory factor analyses were carried out. The results of the Computer Use Survey and the Internet Attitudes Survey were applied independently to both samples as the main components of factor analyses using Varimax rotation (factor loadings cut-off: 0.25) Six factors that were similar in both samples were found through the four studies.
The Computer Use Survey's P1, P2, P3, and P4 categories, which account for 31.69% of the overall variance in the online sample and 31.92% in the offline sample, are as follows:
• SUBSCALE P1—Internet Abuse (Items 3, 5, 6, 7, 9, 12, 20, 24, 25, 38, 41, 42, 43, 44, 55, 65, 66, 67, 68, and 70; in both samples, all of these items have a factor loading > 0.25) This subscale included questions about medical illnesses brought on by heavy Internet use, underlying related dietary changes, fewer hours of sleep due to being online at night, arriving late for appointments, and disregarding one's social and family life.
• Interpersonal conflicts SUBSCALE P2 (items included 27, 35, 40, 45, 46, 47, 48, 63; in both samples, all of these items had a factor loading > 0.25) This
subscale highlighted the existence of significant interpersonal disputes. Along with the inevitable decision to structure the entirety of social life in accordance with the Internet, a propensity for developing virtual connections frequently manifests itself.
• SUBSCALE P3—Introversion/Extroversion (Items included 8, 12, 13, 15, 21, 27, 43, 54, 58, and 72; in both samples, all of these items had a factor loading greater than 0.25) Items pertaining to the introversion/extroversion axis were collected for this subscale. It concentrated in particular on how the individual interacted with others online.
• SUBSCALE P4 — Uncomplicated Internet Use (Items 1, 2, 6, 10, 16, 18, 28, 49, 53, 61, 62, 64, 70, 71; in both samples, all of these items have a factor loading > 0.25): This subscale, which had multiple questions referring to both professional and instrumental net use, isolated an apparent absence of difficulties with Internet use.
The Internet Attitudes Survey's W1 and W2 categories, which account for 38.53% of the overall variance in the online sample and 41.93% in the offline sample, are as follows:
• SUBSCALE W1—Virtual relationship: Using the Internet as an alternative communication medium, virtual relationships are created with the goal of having fun.
• SUBSCALE W2—Professional Internet Use: It portrays the concept of a practical Internet use, emphasizing online purchasing and information searches.
Despite the fact that the six-factor structure was clearly visible in both samples, four of the six online subscales had item loadings that were different from those found in the corresponding offline structure. This implies that the structures are not identical, despite being highly similar. The level of internal reliability of the two questionnaires and the six subscales was then examined using Cronbach's Alpha. In both samples, the two tests yielded satisfactory Alpha levels, with slightly lower results for the online sample:
• Online sample an of the Computer Use Survey was 0.75 while offline sample a was 0.83
• The Internet Attitudes Survey was 0.74 while offline sample a was 0.84.
With levels of Alpha always higher than 0.5 in both samples, we discovered
the same trend in the subscale analyses:
• P1, on-line sample a = 0.88; off-line sample a = 0.90;
• P2, on-line sample a = 0.6; off-line sample a = 0.75;
• P3, on-line sample a = 0.56; off-line sample a = 0.59;
• P4, on-line sample a = 0.51; off-line sample a = 0.59;
• W1, on-line sample a = 0.85; off-line sample a = 0.87;
• W2, on-line sample a = 0.88; off-line sample a = 0.90.
The frequency of pathological actions
Finally, we made an effort to confirm the prevalence of two samples of harmful Internet-related behaviors using the questionnaire data. Unfortunately, the two questionnaires' creators failed to provide a cut-off value in their publications. Given the lack of agreement over the "Internet addiction" phenomenon and the variations in Web usage we discovered amongst the samples, we conducted a traditional psychometric analysis of the data using z-scores. Z-scores measure the position of each point in a normal distribution in terms of its distance above and below the mean in units of standard deviations. They are derived by subtracting the mean from the subject's score and dividing that result by the standard deviation.
We specifically determined cut-off levels for each sample using one (z = 1) and two (z = 2) standard deviations over the mean of the totals for the corresponding group:
• Unproblematic Internet behavior (Level 1) was defined as behavior that was less problematic than 1,
• At risk Internet behavior (Level 2) was defined as behavior that was problematic but not at risk,
• Problematic Internet behavior (Level 3) was defined as behavior that was problematic but not at risk.
Tab L.l 4. I NCIDENCE OF R£KL£V£LIH EACH SUB«:all
Online Offline
subscolr ¡eve} Mi'rt worm1« a Mi'ii №l№N jj
m 3 5 6 11 (5.4%) in 5 15 (7.4%)
2 5 5 10 16 4 20
1 s9 93 182 102 65 167
F2 3 1 3 4(2%) 7 0 9 (4.25%)
2 14 29 121 2 123
1 S4 h6 170 0 72 72
P3 3 5 1 6(3%) 4 2 6 (3%)
2 7 9 16 19 11 30
1 87 94 183 1(15 61 166
P4 3 9 3 12(5.9%) 9 1 10(5%)
2 16 7 23 13 3 16
1 74 94 168 106 70 176
wl 3 8 0 8 (3.9%) 0 0 0
2 21 9 30 (14.8%) 23 11 34 (16.H%)
1 70 95 165 105 63 168
W2 3 3 1 4(2%) 0 0 0
2 17 4 21 (10.3%) 22 15 37 (1 fi.3%)
1 79 99 17S 1(]6 59 165
Table 4 illustrates the distribution of the three levels within the subscales for the two samples.
First, we checked to see if there were any gender disparities within the levels of the subscales in each sample. In the online sample, we didn't see any differences. The sole significant difference in the offline sample was identified in
subscale P2 (Chi-square: 18.42, d.f. = 2, p 0.0001), which shows that there are more interpersonal conflicts in the male group.
Then, we looked for any discrepancies between the two samples. The P2 (Chi-square = 28.22, d.f. = 2, p 0.00001), W1 (Chi-square = 8.27, d.f. = 2, p 0.016), and W2 (Chi-square = 8.9, d.f. = 2, p 0.012) subscales of the data revealed significant differences between the two samples.
Even while the offline group exhibits a higher percentage of at-risk behaviors, we only discovered a small percentage (2-4%) of problematic behaviors in the online sample for the W1 and W2 subscales—more virtual relationships linked to a professional Internet use. In the offline sample, we discovered more risky and problematic behaviors associated with interpersonal disputes in relation to subscale P2. Finally, we looked for interactions between gender and samples using log-linear analysis. No notable findings were made.
DISCUSSION
In this work, we compared web-based evaluation approaches with conventional paper-based procedures to examine the potential use of Internet technologies in psychological research. We specifically contrasted the outcomes of a paper-and-pencil survey administered to a sample of college students with the outcomes we discovered after publishing the same survey online. We conducted a two-level analysis. By contrasting the online assessment tools with their paper-based counterpart, we first confirmed the validity of the online assessment tools. In fact, multiple researchers found a variety of possible issues with the validity of online assessments, including:
• language and cultural barriers,
• a lack of control over the testing environment,
• the probability of unforeseen or transient factors influencing results,
• the possibility of interactions between the constructs being tested and the features of the testing medium.
Our results demonstrated that completing two attitudes and behavior questionnaires online had no effect on the participants' psychometric traits. Internal reliability and the factorial structure did not show any appreciable differences. We specifically discovered the following six subscales in both samples:
• SUBSCALE P1 - Internet abuse;
• SUBSCALE P2 - Interpersonal disputes;
• SUBSCALE P3 - Introversion and extroversion;
• SUBSCALE P4 - Problem-free usage of the Internet.
• SUBSCALE W1 - Online dating;
• SUBSCALE W2 - Business Internet use.
However, some online subscales loaded on items other than those present in the equivalent offline ones if the same structure was clearly discernible in both datasets. This implies that the structures are not identical, despite being highly
similar. These results demonstrate that Web-based data collecting is a reasonable substitute for more conventional techniques and does not statistically improve or decrease response consistency or jeopardize the validity of the test. Given the absence of control over the features of the online sample, this result, which is similar to those observed in Web-based assessments of personality traits and alcohol use, is even more intriguing. The statistics do, however, also show that online and offline versions of the same test can be equal, though they are not always the same. Because of this, it is probably preferable to reevaluate the reliability of a traditional assessment tool before using it online.
Additionally, we contrasted the outcomes of the two samples in an effort to find any appreciable variations. In earlier investigations, other researchers discovered that risk disclosure rates were higher when evaluation techniques that used audio, computers, and videos. The findings of our study, however, do not support this trend. Even though the online sample revealed more risky and problematic interpersonal conflict-related behaviors, it also revealed more problematic virtual connections connected to professional Internet use.
A further explanation for the outcomes of the online sample may be the interaction between the variables being evaluated and the medium utilized to measure them, given that the two questionnaires' topics are Internet attitudes and behavior. Even though the online sample was not controlled, we generally did not detect significant variations between the response sets of online participants and those of people who completed a paper survey. CONCLUSION
In conclusion, our research indicates that Internet-based testing may be a viable substitute for more conventional paper-based tests. However, psychologists who are interested in conducting online tests should take into account a variety of practical concerns, mostly with sampling control and validity testing.
References:
1. Azar, B. (2000). A Web of research. Monitor on Psychology, 31:42-44.
2. Birnbaum, M.H. (Ed.). (2000). Psychological experiments on the Internet. San Diego: Academic Press
3. Byers, C. (2001). Interactive assessment: An approach to enhance teaching and learning. Journal of Interactive Learning Research 12:359-374.
4. Musch, J., & Reips, U.D. (2000). A Brief History of Web Experimenting. In: M.H. Birnbaum (Ed.), Psychological Experiments on the Internet San Diego: Academic Press, pp. 61-87.
5. Pratarelli, M.E., & Browne, B.L. (2002). Confirmatory factor analysis of Internet use and addiction. Cyberpsychology& Behavior 5:53-64.
6. Riva, G. (2001). The mind in the Web: Psychology in the Internet age. CyberPsychology & Behavior 4:1-6.
7. Riva, G. (2001). The mind over the Web: The quest for the definition of a method for Internet research. CyberPsychology and Behavior, 4:7-16.
