CC BY-NC-ND
225
The Role of Engagement in the Development of Critical Thinking in Undergraduates
I. Shcheglova, Y. Koreshnikova, O. Parshina
Received in Irina Shcheglova
March 2018 Junior Researcher, Center of Sociology of Higher Education; Graduate Student, Institute of Education, National Research University Higher School of Economics. Address: 20 Myasnitskaya St, 101000 Moscow, Russian Federation. Email: ishcheglova@hse.ru Yuliya Koreshnikova Analyst, Graduate Student, Institute of Education, National Research University Higher School of Economics. Address: 20 Myasnitskaya St, 101000 Moscow, Russian Federation. Email: koresh-nikova@hse.ru Olga Parshina
Graduate Student, Graduate Center, City University of New York. Address: 365 5th Avenue, New York, NY10016. Email: oparshina@gradcenter.cuny.edu
Abstract. This study explores the link between academic research, extracurricular engagement and the develop-
ment of critical thinking of undergraduate students using a single statistical model. Empirical basis of the research was provided by the results of the Student Experience in the Research University (SERU) survey conducted in one of Russian national research universities in 2017 (N=3,344). Binary logistic regression reveals a statistically significant relationship between the development of critical thinking and student engagement in learning, research and extracurricular activities, higher involvement corresponding to better critical thinking skills. The findings may be useful for developing curricula, allocating student workload, and devising new initiatives for university students.
Keywords: critical thinking, student engagement, student experience, academic activities, extracurricular activities, undergraduate research.
DOI: 10.17323/1814-9545-2019-1-264-289
Translated from Russian by I. Zhuchkova.
Critical thinking is one of the most discussed learning outcomes in higher education of the 21st century. Researchers define this skill as reasonable, purposeful thinking that include analyzing, synthesizing, and evaluating information to make further inferences and decisions [Halpern 1993]. Along with creativity, collaboration, and problem solving, critical thinking is classified among the most in-demand higher-order thinking skills, or 21st century skills [Lai, Viering 2012; Vasi-lyev et al. 2015; Podolsky, Pogozhina 2016; OECD2017]. The role of critical thinking became especially prominent in the digital transformation era, as ubiquitous expansion of the Internet into everyday life made the use of personal portable communications devices a world-
wide trend1. Indeed, it is the ability to critically evaluate the incoming information that enables individuals to make decisions in their career, personal and social life [Strayhorn 2008; OECD2017].
An important trend in the global labor market today is the growing number of jobs that require non-routine, or higher-order thinking skills [Casner-Lotto, Barrington 2006; Podolsky, Popov 2014; Vasily-ev et al. 2015; Dvorkin 2016; Gray 2016; Mikidenko, Storozheva 2017; Froumin, Sorokin 2018]. In particular, this trend manifests itself in employer surveys, which reveal that college transcripts alone are not enough anymore to evaluate candidates' knowledge, skills and productivity [Association of American Colleges & Universities 2018; Podolsky, Popov 2014; Podolsky, Pogozhina 2016]. The vast majority of employers consider critical thinking a fundamental requirement for employment [Casner-Lotto, Barrington 2006; Podolsky, Pogozhina
2016]. The modern economy is in demand of professionals who not only possess knowledge but also know how to apply it in any life situation [Kapuza et al. 2017]. That is why critical thinking is emphasized as one of the key learning outcomes in a number of educational systems and has recently received greater attention from instructors, method-ologists, and education policymakers.
In the early 2000s, looking at the latest PISA2 assessment results, the participating countries realized the need to revise their educational systems and made it their policy to develop critical thinking in school students [Hautamaki 2014; Schleicher 2014; Kapuza et al.
2017]. As noted by Andreas Schleicher, the coordinator of PISA, "The modern world no longer rewards you just for what you know. Google knows everything. The modern world rewards people for what they can do with what they know" 3 using creativity and critical thinking as tools. However, it is not only the school educational systems of the countries seeking intensive development that want to lay the foundation of critical thinking; this skill has become an integral part of higher education models. The Information Notice of the Ministry of Education and Science of Russia No. 05-735 On Enhancing the Federal State Education Standards and Developing the Guidelines for Secondary Education Programs of March 23, 2017 states that Bachelor's degree holders must possess the universal competencies of systems and critical thinking, which include the ability to find, critically evaluate and
1 Brand Analytics. Social Networks in Russia, Winter 2015-2016. Figures, Trends & Perspectives. https://bloq.br-analvtics.ru/socialnve-seti-v-rossii-zima-2015-2016-cifry-trendy-prognozy/; Kemp S. Digital in 2017 Global Overview. https://bloq.hootsuite.com/social-media-statistics-for-so-cial-media-managers/
2 Programme for International Student Assessment. http://www.oecd.orq/pisa/
3 Schleicher A. (2017) What Are the Keys to a Successful Education System. https://www.npr.org/templates/transcript/transcript.php7story-Id=541644277
synthesize information and apply a systematic approach to problem solving4. However, despite the fact that the higher-order thinking skills requirement is considered at the level of national learning standards, no recommendations have been provided so far on the most effective methods and tools to develop these skills. Therefore, studies aiming to identify and develop the tools to improve students' critical thinking skills are in the spotlight in pedagogy and sociology of education now.
The majority of the studies assess the effectiveness of specific teaching practices, collaborative and cooperative learning, and various classroom activities designed to develop and improve critical thinking skills [Halx, Reybold 2005; Shakirova 2006; Muryukina, Che-lysheva 2007]. A number of articles, for instance, reveal a positive correlation between undergraduates' critical thinking skills and their involvement in debates, critical analysis, and teamwork [Smith 1977; Gibson 1985; Astin 1993; Tsui 1999; Coates 2009; Haskell 2016]. Modern western teaching practices, which Russia has recently adopted, are based on the famous Bloom's taxonomy of educational objectives [Bloom 1956]. In the revised version of this taxonomy, critical thinking can be advanced through six fundamental levels of information synthesis: remembering, understanding, applying, analyzing, evaluating, and creating. This approach has proved effective in enhancing students' mastery of material in a discipline-specific manner [Crowe, Dirks, Wenderoth 2008; Gilboy, Heinerichs, Pazzaglia 2015]. However, the primary focus of the taxonomy is on the teaching methods and activities that instructors use in the classroom to improve academic engagement of students. Meanwhile, the taxonomy ignores the potential of university environment that offers opportunity for both academic and non-academic student involvement. However, research confirmed that engagement in research projects, student organizations, and extracurricular events could enhance students' learning outcomes, including higher-order cognitive skills [Astin 1984; Pas-carella, Terenzini 2005; Strauss, Terenzini 2007]. To the best of our knowledge, there are no studies that evaluate the cumulative effects of all the three types of engagement on the development of critical thinking skills in Russian or international literature. The current project thus seeks to explore how academic, research and extracurricular engagement affects the development of critical thinking in undergraduates using a single statistical model. The study addresses the following research questions:
1. Is academic engagement of undergraduates related to their critical thinking skills?
2. Is research engagement of undergraduates related to their critical thinking skills?
4 http://fgosvo.ru/fgosvo/142/141/16
3. Is extracurricular engagement of undergraduates related to their critical thinking skills?
4. What are the cumulative effects of student engagement in various aspects of university life on the development of critical thinking?
Strategies for Critical thinking does not develop spontaneously; it requires a well-or-Developing Critical ganized learning process [Halpern 1993; Popova 2013]. Depending on Thinking Skills the structure of teaching practice, two different approaches to critical thinking instruction are distinguished: embedded and explicit. In the embedded instruction mode, teachers infuse critical thinking using their subject material, while explicit instruction implies specialized courses targeted exclusively on critical thinking skills.
Diane Halpern and Lisa M. Marin [2011] point out that dedicated critical thinking courses are more effective than embedded instruction. Such courses were also found to be more appropriate for students with high academic achievement, as low-achieving students may find themselves struggling with tasks that require higher-order cognitive skills [Zohar, Dori 2003]. Critical thinking courses can be integrated into the curriculum, if possible, or delivered as a supplementary class after regular class hours.
Critical thinking is interpreted as the ability to define a problem, interpret and explain ideas, evaluate arguments, make decisions and inferences, etc. [Glaser 1941; Ennis 1987]. Such cognitive processes can be learned through cooperation in groups [Plotnikova 2015; Johnson, Johnson, Smith 2014], brainstorming [Fahim, Eslamdoost 2014], ad hoc problem solving in the classroom [Popova 2013], and other teaching strategies. There has been no scientific evidence of a single teaching strategy being more effective than any other. Some authors insist on discussion, instructor/student interaction and case studies as the best teaching methods to promote critical thinking [Staib 2003]. Others recommend using real-world examples in teaching to increase the chances for the acquired skills to be applied beyond the classroom settings [Sternberg 2001]. At the same time, researchers point out that most educators focus on transferring established and a priori knowledge or the content matter rather than the instructing techniques to foster critical thinking and analytical skills in students [Fahim, Eslamdoost 2014].
The Effects of Undergraduate Engagement on the Development of Critical Thinking
In the late 1990s, the concept of student involvement was one of the most discussed in the debate on the U.S. system of higher education. It was introduced by Alexander Astin, a professor at the University of California, who postulated that "student involvement reflects the amount of physical and psychological time and energy the student invests in the educational process" [Astin 1984]. The term was introduced to Russian sociology of education in the mid-2010s [Ma-
loshonok 2014], yet studies in this area are rather few. The main idea of student involvement theory is that curriculum should be developed in such a way so as to allow students to invest a sufficient amount of effort and energy into developing the necessary skills [Astin 1984].
Following Russian and international researchers [Astin 1984; Pas-carella, Terenzini 2005; Maloshonok 2014], we identify the following types of undergraduate engagement within the framework of this study:
• Academic engagement— student engagement in the classroom which is assessed based on contribution to a class discussion; application of disciplinary knowledge in a global context; hours spent studying; and out-of-class activities, such as studying with a group of classmates outside of class, communicating with the instructor outside of class about issues and concepts derived from a course, etc.;
• Research engagement— participation in research projects, science research workshops and conferences beyond the curriculum;
• Extracurricular engagement— participation in student organizations.
Empirical evidence shows that these types of engagement are related to the development of higher-order thinking skills to different extents [Centra, Rock 1971; Pace 1984; Astin 1984; Pascarella, Terenzini 2005; Strauss, Terenzini 2007]. Students involved in the learning process are considerably less likely to withdraw than their academically disengaged peers [Kuh 2009; Terentyev, Gruzdev, Gorbunova 2015]. Besides, active participation in university academic life improves undergraduates' self-reported gains, learning satisfaction, academic performance, and persistence [Pascarella et al. 2010]. Frequency of student-faculty and peer interaction in the classroom was found to be positively related to critical thinking skills [Terenzini, Pascarella 1978; 1980; Endo, Harpel 1982, 1983; Pace 1984; Terenzini, Wright 1987; Baxter Magolda 1987; Ory, Braskamp 1988].
Involvement of undergraduate students in research activities can be fostered by professors sharing their research experience in the classroom as well as through guided or self-guided student research. Research engagement is an important factor affecting the development of higher-order thinking skills, including critical thinking [Terenzini, Pascarella 1980; Zydney et al. 2002; Kim, Sax 2009; Miller, Ry-cek, Fritson 2011; Hand et al. 2011]. Participation in research projects was also found to help students develop their research interests and encourage them to pursue postgraduate studies and build academic careers in the future [Russell, Hancock, McCullough 2007].
Along with studies indicating the importance of academic and research engagement for the development of higher-order thinking
Figure 1. The Conceptual Framework of Critical Thinking Development
Student characteristics:
• Gender
• Year of Study
• Academic achievement
• Employment
■3»- Academic ... experiences
Extracurricular experiences
Educational outcomes: Critical thinking
J
skills in undergraduates, more studies emerge that stress the need to engage students in extracurricular activities, such as student clubs and organizations [O'Brien 1995; Strauss, Terenzini 2007]. Participation in intellectual competitions, professional communities and associations correlates positively with analytical skills [Strauss, Terenzini 2007]. Out-of-class experiences contribute to such learning outcomes as teamwork skills, critical thinking, individual and collective responsibility [Pace 1984; O'Brien 1995; Pascarella, Terenzini 2005; Strauss, Terenzini 2007]. Involvement in university non-academic life helps students build their social not only by establishing peer ties but also by connecting to some accomplished academics and business actors [Kasharin 2017; Savelyeva, Voskresensky, Alexandrov 2017]. It has been that extracurricular engagement is a major contributor to starting salary [Hu, Wolniak 2010]. At the same time, a statistically significant negative relationship was found between working more than 20 hours per week and grades, yet working 20 hours or less on campus was significantly and positively related to academic achievement [Pike, Kuh, Massa-McKinley 2008].
All the scholarly articles referred to above focus on one specific type of undergraduate engagement. Studies considering various types of student involvement at once and controlling for the development of higher-order thinking skills, including critical thinking, are extremely rare in the international literature and completely lacking in Russia.
The choice of the conceptual framework for this study was based on the results of previous research that established the relationship between academic performance and undergraduate engagement in: (i) classroom activities, (ii) formal out-of-class instructional experiences, (iii) research, (iv) extracurricular activities. The study relies on the conceptual model proposed by the American scholars Patrick T. Terenzini, Leonard Springer, Ernest T. Pascarella, and Amaury Nora, who were the first to notice that student engagement in various expe-
Table 1. Sample Characteristics
Variable
Gender Male 32%
Female 68%
Year of study 1 42%
2 26%
3 16%
4 16%
Employment On campus 38%
Off campus 52%
Academic Average score
achievement 7.6
riences in the learning environment may promote the development of higher-order thinking skills. Their basic idea is that undergraduate students differ by the level of pre-college preparation, academic achievement, gender, and socioeconomic status, but university environment provides them all with an opportunity to accumulate student experiences by involving actively in various aspects of university life, thereby fostering the development of higher-order thinking skills [Ter-enzini et al. 1995a; 1995b] (Fig. 1).
Data The study was carried out within the framework of the international project Student Experience in the Research University (SERU)5. The sample involved 3,344 Bachelor's degree students enrolled in one of Russia's national research universities in the academic year 2016/17. Participation in the survey was voluntary. Students received the survey invitation in their university email accounts in April 2017. The response rate was 22 percent. Sample characteristics are presented in Table 1.
In terms of academic achievement, the sample differs from the statistical population by 0.3 scores, the overall average score being 7.3 in 2016/17. The sample is skewed towards females, as female students account for 60 percent of total student population. The resulting gender bias of 8 percent is compensated for through weighting adjustments.
The SERU project includes a student survey supplemented with administrative data. The survey collected information on the level of
5 Information on the SERU Consortium is available at https://ioe.hse.ru/seru/ and https://cshe.berkeley.edu/SERU
undergraduates' skills, including critical thinking, their engagement in different types of experiences, and other characteristics. The data used for analysis was fully anonymized and aggregated.
Changes in critical thinking ability were measured based on students' responses to the question, "Please rate your level of proficiency in analytical and critical thinking skills when you started the program in the university vs. now." The response categories were presented on a six-point scale, ranging from "Very poor" to "Excellent". Student self-assessment of improvement in their critical thinking abilities was estimated as the difference between the two variables obtained from the responses, which served as the basis for a dichotomous variable describing how students' critical thinking skills changed since beginning university, "Did not change" (36%), and "Improved" (64%). This measure thus represents retrospectively self-assessed development of critical thinking.
The respondents were asked how frequently they had engaged in the following: (i) in-class (factor score) and out-of-class (factor score) academic activities; (ii) research (participation in research projects: No=0; Yes=1); and (iii) extracurricular experiences (participation in student organizations (No=0; Yes=1) (Appendix А).
The dataset collected during the survey was complemented with administrative information, including such student characteristics as gender (Male=0; Female=1), year of study (1; 2; 3; 4), employment on campus (No=0; Yes=1) or off campus (No=0; Yes=1), and academic achievement (average score, 1-10) calculated as the ratio of the sum of all grades to the number of disciplines, no allowance being made for non-attendance of any kind. Those indicators were used as control variables in the statistical model. The questions about paid employment (including internships) on and off campus yielded the rates of 38 and 52 percent, respectively. However, it appears to be impossible to verify the type of students' employment and the exact number of hours they spent working.
Data Analysis The direct effects of student engagement on the development of crit-Strategy ical thinking were assessed using the method of binary logistic regression, which allows testing the direction and strength of the relationships between the dichotomous dependent (predicted), variable and various independent variables, (predictors), as well as measuring the contribution of individual predictors to the model. Students' self-reported level of proficiency in critical thinking was the predicted variable. The key independent variables were represented by indices of academic (in-class or out-of-class), research and extracurricular engagement of undergraduates while controlling for such factors as gender, year of study, academic achievement, and employment.
Indices of in-class and out-of-class academic engagement obtained by factor analysis are given in Appendix B. Cronbach's alpha
Figure 2. Self-Reported Level of Critical Thinking Skills among Undergraduates Who Engaged or Did Not Engage in Research While Being Students
69%
59%
41%
31%
Did not change
Engaged in a research project Yes No
Improved
Figure 3. Self-Reported Level of Critical Thinking Skills among Undergraduates Who Engaged or Did Not Engage in Extracurricular Activities (Student Organizations) While Being Students
31%
Did not change
69%
62%
Engaged in a student organization Yes No
Improved
was used to verify internal consistency of the items on the scales of in-class and out-of-class academic engagement. The estimated Cron-bach's alpha coefficient was 0.82 for the in-class academic engagement scale and 0.71 for the out-of-class academic engagement scale, which indicates high internal consistency for the two indices.
Figures 2 and 3 present the descriptive statistics for the changes in critical thinking skills based on the subjective perceptions of undergraduates who engaged or did not engage in research (Fig. 2) and student organizations (Fig.) while being students.
As Figures 2 and 3 show, improvement in the level of critical thinking skills is reported more often by students engaged in research and extracurricular activities than by their disengaged peers. Correlation analysis reveals a weak yet statistically significant positive correlation between improvement in the level of critical thinking skills and in-class academic engagement (r=0.15) at significance level p<0.001, correlation with involvement in out-of-class academic experiences being non-significant.
Analysis Results Table 2 presents the results of binary logistic regression. The resulting model allows for correct classification of 67 percent of the respondents. Skewness of the model was tested using the mean value of the unstandardized residuals (M=0.0; confidence interval [-0.19; 0.19]). Homoscedasticity test did not reveal any statistically significant correlations between the residuals and the predictors, hence the data is ho-moscedastic. A test for correlations among the variables selected for
Table 2. Binary Logistic Regression Analysis Examining the Relationship Between Different Types of Undergraduate Engagement and Students' Self-Reported Critical Thinking Skills
Variable Self-Reported Critical Thinking Skills: Did not change=0; Improved=1
Engagement in different aspects of university life B (S.E.) Exp(B) Wald
In-class academic engagement 0.324 (.055) *** 1.382 34.603
Out-of-class academic engagement -0.029 (0.050) 0.972 0.334
Research engagement 0.256 (0.098)*** 1.292 6.792
Extracurricular engagement 0.237 (0.098) ** 1.267 5.804
Student characteristics
Academic achievement (average score) -0.063 (0.046) 0.939 1.847
Gender (female) 0.204 (0.099) 1.226 4.223
Year of study (first year being the reference group)
2 0.508 (.115) *** 1.662 19.379
3 0.837 (.146) *** 2.310 32.802
4 1.245 (.161)*** 3.474 59.733
Employment on campus -0.007 (.138) 0.993 0.003
Employment off campus -0.176 (.109) 0.839 2.590
Constant 0.370 (0.355) 1.447 1.086
Nagelkerke pseudo R2 0.099
Overall predictive accuracy 66.5
*p < 0.05; **p < 0.01; ***p < 0.001
regression analysis found statistically significant correlations among the engagement-related variables (p<0.001), but the strength of correlations does not exceed 0.2. The variance inflation factor (VIF) was used to quantify multicollinearity. The VIF coefficient was found to be in the range [1; 2], which means that there are no significant linear relations among the variables.
Statistically significant coefficients are observed for the following independent variables: "in-class academic engagement" (p < 0.001), "research engagement" (p < 0.001), "engagement in student organizations" (p < 0.01), and "year of study" (p < 0.001). Exp(B) values greater than 1 indicate a positive correlation between the predictor and the predicted variable, so increasing the predictor value will increase the odds of success—in this case, the level of critical thinking skills. The variables "gender", "academic achievement", "employment on campus", and "employment off campus" do not contribute to critical thinking development.
It is important to remember, while interpreting the results, that regression analysis detects the existing correlations among the variables, which do not explain cause and effect. Besides, the full model explains a comparatively low proportion of the variance (Nagelkerke R2 = 9.9%), which implies that some factors are left unattended, such as the nature of research activities (individual or group projects, degree of student autonomy, etc.), family characteristics, or types of extracurricular experiences.
Conclusions and The findings of this study indicate that academic, research and exFuture Prospects tracurricular engagement of undergraduates is positively associated with critical thinking skills. Classroom participation appears to be the strongest predictor of critical thinking, which confirms the necessity to intensify the effective classroom practices. However, academic involvement alone is not enough in the modern world, so universities should also consider other aspects of university life, namely research and extracurricular activities.
An important finding of this study is that it determines the roles of academic and non-academic experiences in the development of critical thinking in undergraduates. Evidence of the crucial role of research engagement is in accordance with the conclusions made by international scholars, who established that participation in scientific events contributes to the development of independent thinking skills and promotes idea synthesis and evaluation [Kinzie 2010; Kilgo, Sheets, Pas-carella 2014], and Russian researchers, who contend that the development of higher-order thinking skills cannot be achieved through mastery of theoretical knowledge alone but requires learner engagement [Mikidenko, Storozheva 2017:371].
Engagement in student organizations was also found to play a significant role in the development of critical thinking, which is confirmed in a number of international studies [O'Brien 1995; Strauss, Terenzini 2007]. Since this study did not control for the type of student organization, the results cannot be extrapolated to specific associations and clubs. Meanwhile, available empirical evidence shows that participation in different types of student organizations yields different outcomes. For example, involvement in political organizations and creative activities has a positive effect on academic performance, whereas participation in sports and religious involvement do not affect student achievement significantly [Baker 2008; Kasharin 2017].
It follows from our findings that academic performance is not related directly to the development of critical thinking. This inference is consistent with the conclusions made by Russian [Podolsky, Pogozhi-na 2016; Rudakov et al. 2017] and international researchers [Cas-ner-Lotto, Barrington 2006; 0ECD2017] who assert that good grades and the "right diploma" are not enough to guarantee that graduates possess all the necessary skills and will make productive employees.
Gender and engagement in paid work are found to be insignificant factors, as opposed to the year of study—the older the students, the higher they estimate their own critical thinking skills, which is supported by previous findings [Halpern, LaMay 2000].
The results of this study point to the need to extend the range of academic and non-academic activities for university students. The findings may be useful to curriculum developers, methodologists, and instructors; they can be used in developing syllabi to allocate student workload so that students would have the time to involve in research projects and non-academic university life. Research on the practices of organizing the academic and extracurricular activities of undergraduates will promote the integration of new strategies into university education, talent detection, and the creation of an effective learning environment conducive to better education outcomes.
We suggest that, by engaging in various aspects of university life, undergraduate students will be able to develop and improve the necessary skills to get the most of their potential in professional, personal and social life. Nevertheless, it should be noted that the cumulative effects of academic, research and extracurricular engagement explain a comparatively low proportion of the variance in the predicted variable, which means that there are other important factors affecting the development of critical thinking. While this study took student participation in research and extracurricular activities into account, it did not look into the types and structure of such activities. Future research should be focused on research and non-academic engagement to find out which types of such engagement stimulate critical thinking the most, and which activity components should be promoted in university education. It is planned to obtain qualitative data from semi-structured interviews with instructors, heads of research laboratories and leaders of research teams, representatives and coordinators of student organizations, and students themselves.
Limitations of Critical thinking skills were evaluated in this study based on student the Study self-assessment. A widespread position in sociology of education is that self-reported learning gains are invalid measures of actual learning gains [Porter 2013]. However, a number of studies show that the retrospective pretest method, which evaluates skills "as you started the program in university" vs. "now", provides a valid assessment of learning outcomes [Thomson 2017; Zilvinskis, Masseria, Pike 2017].
Low response rates are typical of most surveys, including student ones, which may be related to survey fatigue caused by the increased number of surveys [Dey 1997; Porter, Whitcomb, Weitzer 2004; Gru-zdev 2013; Mavletova, Maloshonok, Terentyev 2014]. Relying on experimental data, researchers conclude that a more efficient strategy, instead of chasing high response rates, would be to collect a small set
of data representing the statistical population and focus more on evaluating and using this data [Fosnacht et al. 2017:262].
The gender bias resulting from convenience sampling was compensated for by weighting adjustments.
Since the sample consisted of undergraduates from the same university and the study did not control for university selectivity, type, or size, the strength of correlations between students' engagement in university life and their critical thinking may vary across institutions. Further research is supposed to use an extended sample to include students from universities of other types.
References Ahuna K. H., Tinnesz C. G., VanZile-Tamsen C. (2011) "Methods of Inquiry": Using Critical Thinking to Retain Students. Innovative Higher Education, vol. 36, no 4, pp. 249-259.
Arum R., Roska J. (2011) Academy Adrift: Limited Learning on Our Campuses.
Chicago: University of Chicago. Association of American Colleges & Universities (2018) Employer Survey & Economic Trend Research. Available at: https://aacu.org/leap/public-opin-ion-research (accessed 19 January 2019). Astin A. W. (1984) Student Involvement: A Developmental Theory for Higher Education. Journal of College Student Personnel, vol. 25, no 4, pp. 297-308. Astin A. W. (1993) What Matters in College? Four Critical Years Revisited. Vol. 1.
San Francisco: Jossey-Bass. Baker C. N. (2008) Under-Represented College Students and Extracurricular Involvement: The Effects of Various Student Organizations on Academic Performance. Social Psychology of Education, vol. 11, no 3, pp. 273-298. Baxter Magolda M. B. (1987) Comparing Open-Ended Interviews and Standardized Measures of Intellectual Development. Journal of College Student Personnel, no 28, pp. 443-448. Bloom B. S. (1956) Taxonomy of Educational Objectives. Handbook I: The Cognitive Domain. New York, NY: McKay. Brint S., Cantwell A. M., Saxena P. (2012) Disciplinary Categories, Majors, and Undergraduate Academic Experiences: Rethinking Bok's "Underachieving Colleges" Thesis. Research in Higher Education, vol. 53, no 1, pp. 1-25. Carpi A., Ronan D. M., Falconer H. M., Lents N. H. (2017) Cultivating Minority Scientists: Undergraduate Research Increases Self-Efficacy and Career Ambitions for Underrepresented Students in STEM. Journal of Research in Science Teaching, vol. 54, no 2, pp. 169-194. Casner-Lotto J., Barrington L. (2006) Are They Really Ready to Work? Employers' Perspectives on the Basic Knowledge and Applied Skills of New Entrants to the 21st Century US Workforce. Washington, DC: Partnership for 21st Century Skills.
Centra J. A., Rock D. (1971) College Environments and Student Academic Achievement. American Educational Research Journal, vol. 8, no 4, pp. 623-634. Coates H. (2009) Engaging Students for Success—2008 Australasian Survey of Student Engagement. Victoria: Australian Council for Educational Research. Crowe A., Dirks C., Wenderoth M. P. (2008) Biology in Bloom: Implementing Bloom's Taxonomy to Enhance Student Learning in Biology. CBE—Life Sciences Education, vol. 7, no 4, pp. 368-381. Dey E. L. (1997) Working with Low Survey Response Rates: The Efficacy of Weighting Adjustments. Research in Higher Education, vol. 38, no 2, pp. 215227.
Dvorkin M. (2016) Jobs Involving Routine Tasks Aren't Growing. Available at: https://www.stlouisfed.org/on-the-economy/2016/ianuary/iobs-involv-ing-routine-tasks-arent-growing (accessed 19 January 2019).
Elsen M. G., Visser-Wijnveen G.J., Van der Rijst R. M., Van Driel J. H. (2009) How to Strengthen the Connection between Research and Teaching in Undergraduate University Education. Higher Education Quarterly, vol. 63, no 1, pp. 64-85.
Endo J. J., Harpel R. L. (1982) The Effect of Student-Faculty Interaction on Students' Educational Outcomes. Research in Higher Education, vol. 16, no 2, pp. 115-138.
Endo J., Harpel R. (1983) Student-Faculty Interaction and its Effect on Freshman Year Outcomes at a Major State University. Paper presented at Meeting of the Association for Institutional Research (Toronto).
Ennis R. H. (1987) A Taxonomy of Critical Thinking Dispositions and Abilities. Teaching Thinking Skills: Theory and Practice (eds J. B. Baron, R. S. Sternberg), New York: W. H. Freeman, pp. 9-26.
Fahim M., Eslamdoost S. (2014) Critical Thinking: Frameworks and Models for Teaching. English Language Teaching, vol 7, no 4, pp. 141-151.
Foreman E. A., Retallick M. S. (2012) Undergraduate Involvement in Extracurricular Activities and Leadership Development in College of Agriculture and Life Sciences Students. Journal of Agricultural Education, vol. 53, no 3, pp.111123.
Fosnacht K., Sarraf S., Howe E., Peck L. K. (2017) How Important Are High Response Rates for College Surveys? The Review of Higher Education, vol. 40, no 2, pp. 245-265.
Froumin I., Sorokin P. (2018) Trudnaya doroga [Difficult Road]. Uchitelskaya gazeta, no 20.
Gibson H. W. (1985) Critical Thinking: A Communication Model. Dissertation Abstracts International, vol. 46, no 11, article 3235A.
Gilboy M. B., Heinerichs S., Pazzaglia G. (2015) Enhancing Student Engagement Using the Flipped Classroom. Journal of Nutrition Education and Behavior, vol. 47, no 1, pp. 109-114.
Glaser E. M. (1941) An Experiment in the Development of Critical Thinking. New York: Columbia University.
Gray A. (2016) The 10 Skills You Need to Thrive in the Fourth Industrial Revolution. Available at: https://www.weforum.org/agenda/2016/01/the-10-skills-you-need-to-thrive-in-the-fourth-industrial-revolution/ (accessed 19 January 2019).
Gruzdev I. (2013) Ispolzovanie onlayn-oprosov v universitete [Using Online Surveys in Universities]. Universitas, vol. 1, no 1, pp. 11-22.
Halpern D. F. (1993) Assessing the Effectiveness of Critical-Thinking Instruction. The Journal of General Education, vol. 42, no 4, pp. 238-254.
Halpern D. F., LaMay M.L. (2000) The Smarter Sex: A Critical Review of Sex Differences in Intelligence. Educational Psychology Review, vol. 12, no 2, pp. 229-246.
Halx M. D., Reybold L. E. (2005) A Pedagogy of Force: Faculty Perspectives of Critical Thinking Capacity in Undergraduate Students. The Journal of General Education, vol. 54, no 4, pp. 293-315.
Hand J., Betters C., McKenzie M., Gopalan H. (2011) Increasing Academic Engagement at HBCU's Through the Implementation of an Undergraduate Research Showcase. Mountain Rise, vol. 6, no 3, pp. 1-13.
Haskell N. (2016) Classroom Strategies to Improve Student Self-Efficacy and Learning Outcomes. Available at: https://www.pearsoned.com/im-prove-self-efficacy-learning-outcomes/ (accessed 19 January 2019).
Hautamaki J. (2014) Retsenziya na knigu: Pasi Salberg. Finskie uroki. Chemu mozhet nauchitsya mir na opyte obrazovatelnoy reformy v Finlyandii? [Review of the book: Pasi Sahlberg (2011) Finnish Lessons. What Can the World Learn from Educational Change in Finland?]. Voprosy obrazovaniya / Educational Studies Moscow, no 4, pp. 260-268. doi: 10.17323/1814-95452014-4-260-268.
Hegedus C. M., Knight J. (2009) Student Participation in Collegiate Organizations—Expanding the Boundaries. Available at: http://www.leadershiped-ucators.org/Resources/Documents/Conferences/Lexington/Hegedus.pdf (accessed 19 January 2019).
Hu S., Wolniak G. C. (2010) Initial Evidence on the Influence of College Student Engagement on Early Career Earnings. Research in Higher Education, vol. 51, no 8, pp. 750-766.
Johnson D. W., Johnson R. T., Smith K. A. (2014) Cooperative Learning: Improving University Instruction by Basing Practice on Validated Theory. Journal on Excellence in College Teaching, vol. 25, no 4, pp. 85-118.
Kapuza A., Kersha Y., Zakharov A., Khavenson T. (2017) Obrazovatelnye rezul-taty i sotsialnoe neravenstvo v Rossii: dinamika i svyaz s obrazovatelnoy poli-tikoy [Educational Attainment and Social Inequality in Russia: Dynamics and Correlations with Education Policies]. Voprosy obrazovaniya / Educational Studies Moscow, no 4, pp. 10-35. doi: 10.17323/1814-9545-2017-4-10-35.
Kasharin M. (2017) Svyaz mezhdu uchastiem studentov v studencheskikh organ-izatsiyakh i ikh sotsialnym kapitalom [Relationship Between Participation of Students in Student Organization and Social Capital] (Student Theses). Available at: https://www.hse.ru/edu/vkr/206749166 (accessed 19 January 2019).
Kilgo C. A., Sheets J. K.E., Pascarella E. T. (2014) The Link between High-Impact Practices and Student Learning: Some Longitudinal Evidence. Higher Education, vol. 69, no 4, pp. 509-525.
Kim Y. K., Sax L. J. (2009) Student-Faculty Interaction in Research Universities: Differences by Student Gender, Race, Social Class, and First-Generation Status. Research in Higher Education, vol. 50, no 5, pp. 437-459.
Kinzie J. (2010) Undergraduate Research: High Impact Practice for All Students. Paper presented at Association of American Colleges and Universities Working Conference (Durham, NC). Available at: http://www.aacu.org/meetings/ undergraduate research/2010/resources.cfm (accessed 19 January 2019).
Kuh G. D. (2009) What Student Affairs Professionals Need to Know about Student Engagement. Journal of College Student Development, vol. 50, no 6, pp. 683-706.
Lai E. R., Viering M. (2012) Assessing 21st Century Skills: Integrating Research Findings. Vancouver, BC: National Council on Measurement in Education.
Marin L. M., Halpern D. F. (2011) Pedagogy for Developing Critical Thinking in Adolescents: Explicit Instruction Produces Greatest Gains. Thinking Skills and Creativity, no 6, pp. 1-13.
Maloshonok N. (2014) Vovlechennost studentov v uchebny protsess v rossiys-kikh vuzakh [Student Engagement in Learning in Russian Universities]. Higher Education in Russia, no 1, pp. 37-44.
Mavletova A., Maloshonok N., Terentyev E. (2014) Vliyanie elementov priglash-eniya na uvelichenie otklikov v onlain-oprosakh [The Influence of Invitation's Elements on the Web Survey Response Rates]. Sociology: Methodology, Methods, Mathematical Modeling (4M), no 38, pp. 72-95.
Mikidenko N., Storozheva S. (2017) Rol nauchno-issledovatelskoy raboty v formirovanii «myagkikh» kompetentsiy studentov [The Role of Scientific Research Work in the Formation of Soft Competences of Students]. Busi-
ness. Education. Law. Bulletin of the Volgograd Business Institute, no 4 (41), pp. 366-371.
Miller R. L., Rycek R. F., Fritson K. (2011) The Effects of High Impact Learning Experiences on Student Engagement. Procedia—Social and Behavioral Sciences, no 15, pp. 53-59.
Muryukina E., Chelysheva I. (2007) Razvitie kriticheskogo myshleniya studen-tov pedagogicheskogo vuza v ramkakh spetsializatsii "Mediaobrazovanie". Uchebnoe posobie dlya vuzov [Developing Critical Thinking in Teacher Education Students Majoring in Media Literacy Education. Study Guide for University Students], Taganrog: Kuchma.
O'Brien E. (1995) Extracurricular Participation and Student Engagement. Available at: https://nces.ed.gov/pubs95/web/95741.asp (accessed 19 January 2019).
OECD (2017) OECD Skills Outlook2017. Paris: OECD. Available at: http://www. oecd.org/edu/skills-beyond-school/ (accessed 19 January 2019).
Ory J. C., Braskamp L. A. (1988) Involvement and Growth of Students in Three Academic Programs. Research in Higher Education, vol. 28, no 2, pp. 116-129.
Pace C. R. (1984) Measuring the Quality of College Student Experiences. An Account of the Development and Use of the College Student Experiences Questionnaire. Los Angeles: Higher Education Research Institute Graduate School of Education University of California.
Pascarella E. T., Terenzini P. T. (2005) How College Affects Students. Vol. 2: A Third Decade of Research. San Francisco, CA: Jossey-Bass.
Pascarella E. T., Seifert T. A., Blaich C. (2010) How Effective are the NSSE Benchmarks in Predicting Important Educational Outcomes? Change, vol. 42, no 1, pp. 16-22.
Pike G. R., Kuh G. D., Massa-McKinley R.C. (2008) First-Year Students' Employment, Engagement, and Academic Achievement: Untangling the Relationship between Work and Grades. Naspa Journal, vol. 45, no 4, pp. 560-582.
Plotnikova N. (2015) Formirovanie kriticheskogo myshleniya studentov vuza v usloviyakh komandnoy formy organizatsii obucheniya [Developing Critical Thinking in Undergraduates under the Command Teaching Style], Kazan: Kazan Federal University.
Podolsky O., Pogozhina V. (2016) Klyuchevye kompetentsii vypusknikov i molo-dykh spetsialistov pri prieme na rabotu [Key Competences of Graduates and Young Professionals for Employment]. Science Review: Humanities Research, no 1, pp. 96-103.
Podolskiy O., Popov D. (2014) Pervoe issledovanie kompetentsiy vzrosly-kh v Rossii [The First Assessment of Adult Competencies in Russia]. Vo-prosy obrazovaniya / Educational Studies Moscow, no 2, pp. 82-108. doi: 10.17323/1814-9545-2014-2-82-108.
Popova E. (2013) Formirovanie kriticheskogo myshleniya studentov v kurse dist-sipliny "inostranny yazyk" kak faktor professional'noy kompetentnosti [Development of Critical Thinking Skills within the Foreign Language Course as a Factor of Professional Competence in Undergraduates]. Vestnik MGLU, iss. 8 (668), pp. 167-180.
Porter S. R., Whitcomb M. E., Weitzer W. H. (2004) Multiple Surveys of Students and Survey Fatigue. New Directions for Institutional Research, vol. 121, pp. 63-73.
Porter S. R. (2013) Self-Reported Learning Gains: A Theory and Test of College Student Survey Response. Research in Higher Education, vol. 54, no 2, pp. 201-226.
Rudakov V., Chirikov I., Roshchin S., Drozhzhina D. (2017) Uchis, student? Vli-yanie uspevaemosti v vuse na startovuyu zarabotnuyu platu vypusknikov
[The Impact of Academic Achievement on Starting Wages of Russian University Graduates]. Voprosy Ekonomiki, no 3, pp. 77-102.
Russell S. H., Hancock M. P., McCullough J. (2007) Benefits of Undergraduate Research Experiences. Science, vol. 316, pp. 548-549.
Savelyeva S., Voskresensky V., Alexandrov D. (2017) Rol vneklassnoy aktivnos-ti v formirovanii sotsialnogo neravenstva: sluchay malogo goroda [The Role of Extracurricular Activities in the Formation of Social Inequality: The Case of a Small Town]. Obrazovanie i sotsialnaya differentsiatsiya [Education and Social Differentiation] (eds. M. Carnoy, I. Froumin, N. Karmaeva), Moscow: Higher School of Economics.
Schleicher A. (2014) Opinion: What Asian Schools Can Teach the Rest of the World. Available at: http://edition.cnn.com/2013/12/03/opinion/educa-tion-rankings-commentary-schleicher/ (accessed 19 January 2019).
Shakirova D. (2006) Formirovanie kriticheskogo myshleniya uchashchikhsya i studentov: model i tekhnologiya [Developing Critical Thinking in Students: Model and Technology]. Educational Technology & Society, no 9(4), pp. 284-292.
Smith D. G. (1977) College Classroom Interactions and Critical Thinking. Journal of Educational Psychology, vol. 69, no 2, pp. 180-190.
Staib S. (2003) Teaching and Measuring Critical Thinking. Journal of Nursing Education, vol. 42, no 11, pp. 498-508.
Sternberg R. (2001) Teaching Problem Solving as a Way of Life. Developing Minds: A Resource Book for Teaching Thinking (ed. A. Costa), Alexandria, VA: ASCD, pp. 451-454.
Strauss L. C., Terenzini P. T. (2007) The Effects of Students' In and Out-of-Class Experiences on Their Analytical and Group Skills: A Study of Engineering Education. Research in Higher Education, vol. 48, no 8, pp. 967-992.
Strayhorn T. L. (2008) The Role of Supportive Relationships in Facilitating African American Males' Success in College. Naspa Journal, vol. 45, no 1, pp. 26-48.
Terentyev E., Gruzdev I., Gorbunova E. (2015) Sud idet: diskurs prepodavate-ley ob otseve studentov [The Court Is Now in Session: Professor Discourse on Student Attrition]. Voprosy obrazovaniya/Educational Studies. Moscow, no 2, pp. 129-151. doi: 10.17323/1814-9545-2015-2-129-151.
Terenzini P. T., Pascarella E. T. (1978) The Relation of Students' Precollege Characteristics and Freshman Year Experience to Voluntary Attrition. Research in Higher Education, vol. 9, no 4, pp. 347-366.
Terenzini P. T., Pascarella E. T. (1980) Toward the Validation of Tinto's Model of College Student Attrition: A Review of Recent Studies. Research in Higher Education, vol. 12, no 3, pp. 271-282.
Terenzini P. T., Springer L., Pascarella E. T., Nora A. (1995a) Influences Affecting the Development of Students' Critical Thinking Skills. Research in Higher Education, vol. 36, no 1, pp. 23-39.
Terenzini P. T., Springer L., Pascarella E. T., Nora A. (1995b) Academic and Out-of-Class Influences on Students' Intellectual Orientations. The Review of Higher Education, vol. 19, no 1, pp. 23-44.
Terenzini P. T., Wright T. M. (1987) Influences on Students' Academic Growth During Four Years of College. Research in Higher Education, vol. 26, no 2, pp. 161-179.
Thomson G. (2017) Self-Reported Learning Outcomes and Assessment: Making the Case. Paper presented at 43rd Annual Meeting of the California Association for Institutional Research (Concord, CA, November 2017).
Tsui L. (1999) Courses and Instruction Affecting Critical Thinking. Research in Higher Education, vol. 40, no 2, pp.185-200.
Vasilyev K. et al. (2015) Razvitie navykov dlya innovatsionnogo rosta v Rossii [Developing Skills for Innovative Growth in the Russian Federation], Moscow:
Alex.
Zilvinskis J., Masseria A. A., Pike G. R. (2017) Student Engagement and Student Learning: Examining the Convergent and Discriminant Validity of the Revised National Survey of Student Engagement. Research in Higher Education, vol. 58, no 8, pp.880-903.
Zohar A., Dori Y. J. (2003) Higher Order Thinking Skills and Low Achieving Students: Are They Mutually Exclusive? Journal of the Learning Sciences, vol. 12, pp. 145-181.
Zydney A. L., Bennett J. S., Shahid A., Bauer K. W. (2002) Impact of Undergraduate Research Experience in Engineering. Journal of Engineering Education, vol. 91, no 2, pp. 151-157.
Appendix A. Descriptive Statistics by Variables
Variable Percentage
In-class engagement
Asked an insightful question in class
Never 5
Rarely 20
Occasionally 30
Somewhat often 19
Often 16
Very often 10
Contributed to a class discussion
Never 1
Rarely 8
Occasionally 23
Somewhat often 19
Often 23
Very often 27
Used disciplinary knowledge in a global context
Never 1
Rarely 8
Occasionally 22
Somewhat often 24
Often 27
Very often 18
Found your courses so interesting that you did more work than was required
Never 8
Rarely 24
Occasionally 33
Somewhat often 17
Often 12
Very often 7
Out-of-class engagement
Studied with a group of classmates outside of class
Never 10
Rarely 20
Occasionally 26
Variable Percentage
Somewhat often 19
Often 15
Very often 10
Worked on class projects with classmates outside of class
Never 21
Rarely 15
Occasionally 23
Somewhat often 18
Often 17
Very often 14
Helped a classmate better understand the course material when studying together
Never 6
Rarely 20
Occasionally 32
Somewhat often 20
Often 15
Very often 8
Communicated with the instructor outside of class about issues and concepts derived from a course
Never 16
Rarely 32
Occasionally 28
Somewhat often 12
Often 8
Very often 4
Research engagement
While a student, have you completed or are you now participating in a research project(s)?
Yes 56
No 44
Extracurricular engagement
While a student, have you involved or are you currently involved in a student organization(s)?
Yes 38
No 62
Voprosy obrazovaniya / Educational Studies Moscow. 2019. No 1. P. 264-289
Appendix B. Undergraduate Engagement in Academic (In-Class and Out-of-Class), Research and Non-Academic Aspects of University Life
In-Class Engagement Index Out-of-Class Engagement Index
During this academic year, how often have you done each of the following? 1) Asked an insightful question in class (0.84) 2) Contributed to a class discussion (0.85) 3) Applied disciplinary knowledge in a global context (0.83) 4) Found your courses so interesting that you did more work than was required (0.67) Proportion of explained variance: 59.2% During this academic year, how often have you done each of the following? 1) Studied with a group of classmates outside of class (0.72) 2) Worked on class projects with classmates outside of class (0.55) 3) Helped a classmate better understand the course material when studying together (0.58) Proportion of explained variance: 61.4%
Response categories: Never (1), Rarely (2), Occasionally (3), Somewhat often (4), Often (5), Very often (6)
