CC BY-NC-ND
60
What Is the Impact of the Flipping the Classroom Instructional e-Learning Model on Teachers
Lut De Jaegher
Received in Lut De Jaegher
December 2019 Lecturer, Artevelde University College Ghent (Belgium). Адрес: Artevelde University College Ghent, Hoogpoort 15, BE—9000 Gent. E-mail: lut.dejaegher@ arteveldehs.be
Abstract. Flipping the classroom is an instructional model in which students learn basic subject knowledge prior to the face-to-face class moment, where they can have active learning experiences with their peers and teachers. Research revealed the positive effects for students, who can learn at their own pace, reach up to the highest level of the thinking skills of Bloom's taxonomy, exercise and improve their collaboration, communication and ICT skills. Where most of the research concentrates on the learning effects for the students, this paper presents the results of recent European research on the impact for teachers. Setting up a learning path for flipped classroom, is a big challenge. Together with 7 European partners from Belgium, Italy, Bulgaria, Slovenia, Poland and the Netherlands, we did research on how students and teachers perceive the implementation of the flipped classroom model in their teaching and how challenging the integration of technology in their lessons is.
We also asked about their perceptions: are the benefits worth the efforts, is the flipped classroom model improving their teaching skills and what are their needs and requirements to get succeed? We started by getting the teachers a flipped classroom instruction to learn the method, combined with a face-to-face training in Belgium, where they were supported to create a flipped classroom learning path for their own subjects and classes. Then, they implemented the method in their institutions for adult and higher education. The surveys were conducted after this try out, in all of the 7 European participating countries. The research results of the surveys will be presented and used to make recommendations that increase the chance of a successful implementation of the flipped classroom method. These recommendations were tested and evaluated during flipped classroom trainings at teacher training departments in Belgium and Vietnam. The qualitative test results will also be presented in this paper. Keywords: Flipped classroom, Active learning, ICT integration, technology in edcuation, Bloom's taxonomy, TPACK.
DOI: 10.17323/1814-9545-2020-2-175-203
1. Introduction At all levels of education, teachers are increasingly being challenged to form creative, critically thinking students who are able to absorb,
integrate and apply knowledge at different levels, going from reproducing facts, understanding concepts and researching and using procedures for problem solving, to the metacognitive knowledge required to overview and respond to complex problems in life and society. There's an ever-increasing demand for teachers to pair "content with engaging, experiential, and innovative learning experiences" [Darling-Hammond et al. 2019]. An important question is how teachers can be equipped with the mindsets and the didactical, pedagogical and technological skills required for deeper student learning in order to reach the 21st century skills, abilities, and learning dispositions.
Keeping up with the rhythm of the rapidly evolving information and communication technology, isn't enough. Research shows the importance of integrating content knowledge, technological knowledge and pedagogical knowledge in teaching. "For this reason, teacher training in information and communication technology (ICT) needs to investigate the theoretical foundations guiding their application and use in the classroom, both at a disciplinary and at a pedagogical level, together with technological knowledge on how ICT work in its implementation." (Rodriguez Moreno, 2019). The Technological, Pedagogical and Content Knowledge (TPACK) framework, designed by Mishra and Koehler [2006] has had a major impact on research and determination of the kinds of knowledge required by teachers in order to integrate ICT in their lessons. In 2019, the TPACK model has had an upgrade by adding "another knowledge domain that teacher must possess to integrate technology in teaching" [Mishra 2019]. Mishra realized that the success of the effort of teachers to integrate technological, pedagogical and content knowledge also depends on their Knowledge of the Context (XK) and how the situational and organizational constrains can effect sustainable change " [Mishra 2019].
The TPACK-model fits perfectly into the taxonomy of Bloom, used since several decades by lot of teachers worldwide to design courses, determine and formulate the expected Student Learning Outcomes (SLO) and to create assessments. The taxonomy of Bloom is based on the classification of thinking skills into 6 hierarchically organized categories, ranged from lower level to higher order. The two-dimensional hierarchical table was first filled with nouns [Bloom 1956] and after revision by Anderson and Krathwolh in 2001, the nouns were changed into verbs [Anderson, Kartwohl 2001].
In 2006 Rex Heer from Iowa State University has again redesigned the taxonomy, into a 3-dimensional framework. The cognitive domain is hereby defined as the intersection of the Cognitive Process and the Knowledge dimension going from concrete (factual, conceptual, procedural) to abstract (metacognitive). The model allows teachers to formulate learning objectives for deep learning at the pace and taking into account the possibilities of the students as group and as individual learners.
Figure 1: Revised version of the TPACK-model
[Mishra 2019]
rov .-■
,...-■" The Technological, Pedagogical and Content Knowledge (TPACK)
/
Technological pedagogical knowlege (TPK)
Pedagogical knowlege (PK)
Technolgical knowlege (TK)
Technolgical content knowlege (TCK)
Content knowlege (CK)
Pedagogical content knowlege (PCK)
Figure 2: Revised version of Bloom's taxonomy [Bloom, 1956]
1956
2001
Evaluation Synthesis Analysis Application Comprehension Knowledge
Create Evaluate Analyse Apply Understand Remember
Noun
to Verb Form
An instructional model or an instruction method that combines TPACK with the ability of describing and achieving lesson objectives spread over the 3-dimensional framework of Bloom's taxonomy, is flipped classroom as part of blended learning. This innovation in teaching and learning using ICT, popped up in diverse educational settings during the first decade of the 21st century, when educa-
Figure 3: The 3-dimensional framework of Bloom's taxonomy [Teach Tought Staff 2016]
Create
an innovative learning portfolio
Metacognitive
Knowlege of cognitions in general as well as awareness and knowlege of one's own cognition
Create
Put elements together to form a coherent whole; reorganise into a new pattern or structure
Procedural
How to do something methods of inquiry, and criteria for using skills algorithms, techniques and methods
Conceptual The interrlationships among the basic elements with a larger structure that enable them to function together
Analyse
Break material into constituent parts and determine how parts relate to one another and to an overall structure or purpose
Make judgements based on criteria and standards
Factual
The basic elements students must know to be acquainted with a discipline or solve problems in it
Remember
Retrrieve relevant knowlege from long-term memory
Apply
Carry out or use a procedure
Understand in a given situation
Construct meanings from instructional messages, including oral, written, and graphic communication
A statement of a learning objective contains a verb (an action) and an object usually a noun)
The verb generally refers to [actions associated with] the intended cognitive process.
The object generally describes the knowlege students are expected to acquire or construct. (Anderson and Krathwohl, 2001, pp. 4-5)
Anderson L. W. (Ed.), Kartwohl D.R. (Ed.), Airasian P. W., Cruikshank K. A., Mayer R. E., Pintrich P. R., Raths J., & Wittrock M. C. (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Education Objectives (Complete edition) New York: Longman.
Model created by: Rex Heer, Iowa State University Center for Excellence in Learning and Teaching. Updated January 2012.
Licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License. For additional resources, see: www.celt.iastate.edu/teaching/RevisedBlooms1.html
In this model, each of the blocks shiws an example of a learning objective that generally correspond with each of the various combinations of the cognitive process and knowlege dimensions.
REMEMBER: these are learning objectives—not learning activities. It many usefull to think of preceding each objective with something like: "Students will be able to..."
http://vo.hse.ru/en/
Figure 4: Flipped Classroom instructional model
Before class During class After class
Students continue checking for understanding of the concept through higher order application and evaluation
o
V)
.2 Students explore new
= concepts through learning
activities, including peer discussions and 1 : 1 interactions with the teacher
Students get acclimated with new concept and terminology via digital media. Students may take notes and jot down questions for futher discussion
tors experimented with shifting the lesson instruction from a teacher-centered to a learner-centered model using a range of technology and social and digital media, by offering the lesson content in several forms (video, text, podcasts...) to be studied outside the classroom. Flipped learning as 1 of the many forms of blended learning allows teachers and students to explore the deeper knowledge dimensions inside the classroom, because the basic knowledge a student needs is already reached in anticipation of the in-class face-to-face session. The teaching and activities during the class time include a wide area of active learning types and the possibility of a more personalized interaction between teacher and students on one hand and between students mutually through peer-instruction on the other hand.
As mentioned the flipped classroom method is one of the blended learning types. The differences between both is that blended learning can be defined as the overall pedagogical approach where learning at distance and learning in a face-to-face setting are mixed in several ways whereas flipped classroom is a instruction method where the homework and face-to-face learning are flipped: homework comes first, in-class session next.
In this article we start with a short overview of the most recent research results of the method on learners. We will also formulate ideas for issues that are still missing in the current research, for example the long-term effects on learners. In a second part we will focus on the other important stakeholder often forgotten in research: the teacher.
2. Effects of the Flipped classroom method for learners
2.1. Survey results
Since the launch of the flipped classroom method, a growing body of research has revealed that flipped learning can have a number of positive effects on students [Tomas et al. 2019]. The observation that students show more commitment [Fulton 2012], that they have a more positive attitude towards this way of learning, appreciate the flexibility in learning at their own pace and the possibilities for a differentiated approach, appears in various research reports. This was also confirmed in the research the author of this article recently was involved in, during the European Erasmus+ projects iFlip [iFLIP Project 2017] and FlippingFirst [Flipping First Erasmus+ Project 2017]. Also the fact that the method enhances education access and leads to learning successes for both minority and non minority students [Dziuban et al. 2018] can be underlined as an important benefit. It can be observed that the majority of these studies focus on students in adult education, more specific at higher education (bachelor and master) level.
In the iFlip Erasmus+ project we generated quantitative and qualitative surveys with a target group of adult learners, aged 16 or older, from secondary or higher education level. Together with the 6 project partners from the Netherlands, Belgium, Slovenia, Bulgaria, Italy and Poland, we reached 220 respondents (n=220), which we questioned in a pre-test about their interests and needs for learning. We analysed the results quantitavely, with the use of descriptive statistics. The results were put in a spreadsheet and visually presented with graphics1.
The gender distribution shows 27% males and 73% females. The age distribution provides sufficient responses in all major age groups. However, calculated correlation reveals no significant dependencies between age and learning factors on which the survey is focused.
Fifty percent of the respondents have a higher education (Bachelor, Master, or Ph.D) level. Thirty-one per cent have vocational education and training—at secondary education level combined with a vocational degree, or at post-secondary vocational training level.
As pre-test the respondents were asked to scale 14 learning factors on a Likert scale (Table 1)
1. fully agree
2. somewhat agree
3. neutral
4. somewhat disagree
5. fully disagree
There were 11 factors which have a nominal scale value of 1.00+ and a distinct consensus in the answers (Table 2)
According to research reports on flipped classroom the learning factors mentioned as important by the pre-test respondents are corre-
http://proiectiflip.eu/en/alnar-surveys/
Figure 5: Educational attainment level of respondents, all countries (n = 219, %)
ower than secondary ed.
sponding to what has been reported as benefits of flipped classroom by respondents in the iFlip- and other research reports in post-tests [Tomas et al. 2019; Rodriguez Moreno, Agreda Montoro, Ortiz-Colon 2019; Dziuban et al. 2018; Nurul et al. 2018; Flipping First Erasmus+ Project 2017]. The flipped classroom method has a positive impact on the learner satisfaction, which is an important factor to stay motivated during the learning process. Concerning the learner achievements though, no recent study proves a signifant impact difference in favor or disfavor of flipped classroom [Sommer, Ritzhaupt 2018]. But, as Robert Talbert concludes in his opinion article "What does research say about flipped learning?" [Talbert, 2018], some important questions need to be asked to properly frame the results:
• Is the flipped course an introduction to the subject, or advanced?
• Is the course for an undergraduate or a graduate target group
• Is a lot of technology involved or not
• Is the course a small section or a large section of the overall content
• Is the whole course flipped or only part of it
• Does the online part uses instruction video's or not
Table 1: Learning factors (full list)
A. I like learning new things
B. I usually learn fast and with ease
C. I like to have control over the learning process
D. I like to take initiative and construct my own learning path, given some guidelines
E. I like lively discussions in class
F. I like to set my own learning pace
G. When in class, I like to sit quietly and listen
H. I prefer to have time to explore and reflect upon new ideas
I. I like to have additional materials and resources along the main training texts/content J. Having too many training content sources upsets me
L. I like sharing my opinion on things I have read, listened to, or seen
Q. When I can't keep to-date with assignments and learning deadlines, I tend to lose motivation for learning
R. I like to set my own learning goals
S. I like to be able to track my own progress and measure achievements
Table 2: Learning factors with nominal scale value of 1.00+
A. I like learning new things
S. I like to be able to track my own progress and measure achievements F. I like to set my own learning pace
H. I prefer to have time to explore and reflect upon new ideas
L. I like sharing my opinion on things I have read, listened to, or seen
C. I like to have control over the learning process E. I like lively discussions in class
D. I like to take initiative and construct my own learning path, given some guidelines
I. I like to have additional materials and resources along the main training texts/content R. I like to set my own learning goals
B. I usually learn fast and with ease
2.2. Conclusion The more flipped classroom and other blended learning methods are learners used and introduced in teaching practice, the better future research will be able to reveal the long term benefits and pitfalls of these learning methods [Nurul et al. 2018; Tomas et al. 2019; iFLIP Project 2017; Flipping First Erasmus+ Project 2017; Teach Tought Staff 2016; Kwan Lo, Foon Hew 2017]. For now the most mentioned advantages (learning direct and indirect outcomes) can be grouped into 3 categories: learning experience at home, learning quality in class and collateral learning effects.
2.2.1. Advantages for Learning experience at home
learners In a flipped classroom setting, learners can have a better learning experience. Having to learn new or review prerequisite content before coming to the face-to-face class, they feel more prepared. The students are able to learn at own pace and without stress, and they have the opportunity to recall and review online lesson content as many times as they want and need.
Learning quality in class
Thanks to the preparation in advance, the students will experience more opportunities to ask questions. Working in teams or listening to each others' presentations can help them to learn from each other using peer-instruction and differentiation. Their over-all learning motivation improves.
Collateral learning effects
Students use hard- and software which will enhance their ICT knowledge. They get use to learn independently and form new learning habits. The students improve their communication skills with peers and teachers.
Research also reveals pitfalls for learners. The most mentioned pitfalls can be related to technique, communication and personality.
2.2.2. Pitfalls for Technical pitfalls
learners Flipped classroom only works well if there are no technical issues. Problems can raise when the internet access is slow or fails, and hyperlinks, video or sound doesn't work properly. The learners need a good ICT knowledge to make sure they can use the online content. Any technical problem can have negative effects on the motivation and the learning experience of the learners.
Sometimes a part of the course content is not suitable for the method.
Communicational pitfalls
Flipped classroom teaching cannot be successfully introduced to the learners without preparing them well about the method, the purposes and the expectations. If students doesn't have the opportunity to communicate with or be coached by their teachers during the out of class learning, they will loose motivation.
Personal pitfalls
Some learners prefer traditional in-class learning, others doesn't have the motivation or the will to complete the priliminary homework.
3. Effects of the Flipped classroom method for teachers
As mentioned by Nurul and Abus [Nurul et al. 2018]: "The success of this method depends on the proper development of the resource materials, delivery methods, assessment strategy, adequate facilities etc. Therefore, proper planning of the educational managers is necessary in order to train the teachers for their mindset change and use more flipped classes then the traditional lecture and to make them competent in developing resources and also to guide the students properly."
3.1. Survey results
One of the major problems teachers using the flipped classroom method encounter, is the considerable workload of creating flipped
learning materials, the need of more ICT knowledge and/or the ability to get support from a technician, and students' disengagement in the out-of-class learning [Flipping First Erasmus+ Project 2017; Teach Tought Staff 2016; Fulton 2012; iFLIP Project 2017; Kwan Lo, Foon Hew 2017; Dziuban et al. 2018].
The team of the Erasmus+ iFlip-project researched the educators' satisfaction on teaching with the flipped classroom method based on 3 research questions:
• Q1: How well is the flipped classroom method known by the educators before starting the program (http://projectiflip.eu/en/pro-ject-results/)?
• Q2a: How do the teachers experience a specific training on the flipped classroom method?
• Q2b: What is the effect of a flipped classroom training of 1 month, of which 5 days face-to-face and the rest online, to train teachers to realize a flipped classroom course for their own teaching practice and share their knowledge with their peers?
• Q3-: How do the educators experience the implementation of the method in their specific teaching settings?
The total number of respondents in the pre-test survey was n=96, spread over all the participating project-partner countries. Gender distribution shows 27% males and 73% females, teaching in adult education of all possible education levels.
Age distribution provides sufficient responses in all major age groups. Almost half (47%) of the respondents fall within two age groups in the 35-44 years range.
Question 1: how well is the flipped classroom method known by the educators before starting the program?
The survey results showed that 18% already used flipping the classroom, and another 23% didn't but found themselves immediately ready to use it ("I know what it is and how to use it"). This brings a combined share of 41% of trainers/educators who would be ready and able to work with Flipped classroom with some assistance on the content part from the iFlip training project. Another 11.5% claimed that they "know what FC is, but not how to use it". Remarkably high shares of respondents (just over 28%) had only heard of the term but didn't know the meaning. They could become interested in Flipped classroom if adequate and sufficient training could be provided both on theory (pedagogical, didactical and technological) and practice. And another share of almost 20% had no clue at all what FC means.
Question 2a: how do the teachers experience a specific "flipped classroom training" on the flipped classroom method
Question 2: what is the effect of a flipped classroom training of 1 month, of which 5 days face-to-face and the rest online, to train teachers to realize a flipped classroom course for their own teaching practice
Figure 6: Trainers'survey, age groups
(combined sample, n = 96, number of respondents)
25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64
4
27
5
13
18
14
10
Figure 7: Trainers'survey, familiarity with the flipped classroom concept in advance
(n = 96, %)
Are you familiar with the flipped classroom concept?
I've no idea what it is 19,8
I use it already 17,7
I've heardof the term, but nothing more 28,1
I know what it is, and how to use it 22,7
I know what it is, but not how to use it 11,5
In preparation of the second part of the research, a flipped classroom pilot course with a 1 month online training and a five-day face-to-face training was conducted in Ghent, Belgium to train 16 adult learners (selected by the participating partners from each country) on creating flipped classroom courses for their particular subjects, starting with the didactical and pedagogical aspect, the use of technology and the creation of a learning path in a Learning Management System. For the purpose of this training we used and taught Moodle as LMS. The FTC methodology was introduced to the participating adult teachers and the opportunities for flipped classroom and blended learning for adult learners via LMS platforms were discussed.
Figure. 8. Pilot for trainers, age Figure. 9. Pilot for trainers, groups (n = 16), % gender (n = 16), %
Male 33,3
Female 66.7
The group of educators who participated at the training in Ghent, returned back to their countries and institutions with 2 tasks: create own pilot courses for their students, and transfer the flipped classroom method to their colleagues via internal know-how sharing sessions. Where necessary, technical assistants supported the teachers during the development of their pilots.
At the end of the training, the educators filled in a questionnaire that was developed in order to assess the piloting courses from teacher's point of view. Since there are only 16 educators in the sample, there is little point in statistical analysis of the questionnaires' responses. We only present highlights of the teachers' feedback, which we tested and evaluated qualitatively using interviews and observations, in July 2018 in the Hanoi National University of Education with 60 professors from 14 different departments and in November 2018 during a training in the Hanoi Pedagogical University 2 with 58 participants from 12 different departments.
The first two questions show an overwhelmingly positive attitude towards the iFlip- flipped classroom training approach. Educators seem to be willing to embrace the method in their practice. Further, their opinion is that the course content is up to the needs and satisfaction of the learners who participated in the pilots.
We used the same set of questions to observe and analyze how the 118 participants at a similar training at the HNUE and HPU2 Universities in Vietnam, taught by 1 of the iFlip researchers was experienced. Each of the 26 departments participated at the structured interviews and feedback sessions. We wanted to understand:
Table 7: Trainers 5-days satisfaction and efficiency survey (n = 16)
very neg neg neutral pos very pos
1 Approach is useful for learning your course content 4 12
2 Course content is satisfying for your learners 1 5 10
3 Was the duration of the 5 days training efficient 6 8 2
4 Was the scope of the training efficient 1 1 6 8
5 Are the learners interested in the methods and instruments 5 11
6 Learners are interested in course materials 1 6 9
7 Learners are interested in course content 4 5 7
8 Learning effect of the method 3 5 8
9.1 I flip material: reader-friendliness 1 9 6
9.2 I flip material: completeness 3 9 4
9.3 I flip material: appropriateness 1 6 9
9.4 I flip material: userfriendlyness 1 6 9
10 Appropriate size of the group 4 12
11 ICT knowhow appropriate to the course 2 4 6 4
12 Room equipment appropriate 2 6 8
13 Would you recommend the iFlip course 6 10
• What conditions must be met by the participants in the training in order to be able to follow the training successfully?
• Which conditions teachers and their teams must meet in order to be able and willing to work successfully with the method
• Which subjects lend better and less well to the use of the method
• Whether the flipped classroom method itself is an efficient way to teach the flipped classroom method to teachers
• How obstacles/pitfalls for the use of the flipped classroom method can be avoided
During the observations, interviews and feedback sessions with the 118 Vietnamese teachers and professors at the start of the face to face sessions, we came to the conclusion that the preparatory phase with online material, studied by the teachers in advance, didn't assure them that they would have enough ICT knowledge to work effectively with the method. They understood and knew the theory and didactics behind the method, but their self-efficacy on ICT matters was weak.
Figure 9 and 10: Q1 and Q2 (n = 16, %)
Q1. Do you find the iFlip approach usefull for learning of your course?
Very
positive
73,3
Q2. Do you think the course content satifying for learners?
Neutral
Very
positive
60,0
Figure 11 and 12: Q3 and Q4 (n = 16, %)
Q3. Efficience of duration of Q4. Efficience of scope of
the course training the course training
Very positive 46,7
Positive 40,0
Positive 13,3
Neutral 40,0
Very
positive
46,7
The efficiency of course duration (Q3) raises some concern with a 40% share of neutral responses. We saw similar hesitation by respondents in some countries when asked the same question. Closer evaluation reveals that this could be a result of the novelty of the method and some insecurity among learners and educators alike on the appropriate and convenient course duration. The efficiency of the training as a whole (Q4) attracts 86.7% positive-side responses.
90% of the Vietnamese respondents were satisfied with the training efficiency, content and duration. After the training 20% of them asked for further support.
Figure 13 and 14: Q5 and Q6 (n = 16, %)
Q5. Methods and instruments used for teaching
Q6. Interest of learners for presented material of your course
Neutral 6,7
Positive 40,0
Very
positive
53,3
Figure 15 and 16: Q7 and Q8 (n = 16, %)
Q7. Interest of learners for Q8. Learning effect of iFlip
presented content of your approach in your course
course
Question 3: how do the educators experience the implementation of the method in their specific teaching settings
We also find very positive responses concerning both the educators' perception of the methods and instruments they have used (Q5), and of the interest shown by the learners for the materials developed for the course (Q6).
The educators evaluated the learners' interest positively, but one quarter of the respondents hesitated and gave a neutral response. We believe this is due to the fact that the method is relatively new and the lack of objective benchmarks (or evaluation of parallel-running groups
Figure 17 and 18: Q9.1 and Q9.2 (n = 16, %)
Q9.1. Use of iFlip materials— reader-frendliness
Very
positive
40,0
Q9.2. Use of iFlip materials-completeness
Figure 19 and 20: Q9.3 and Q9.4 (n = 16, %)
Q9.3. Use of iFlip materials— appropriateness for the needs of adult learners
Q9.4. Use of iFlip materials—user-frendliness for trainers
Neutral
6,7
Neutral
6,7
Positive 40,0
Very
positive
53,3
Positive 40,0
Very
positive
53,3
with included graded learning assessment) prevents the educators from objectively leaning positively or negatively, hence the hesitation. Similarly positive and with a 20% neutral responses is the evaluation of the learning effect of the FTC method.
The educators were also asked to give their opinion on the use of training materials developed for the iFLIP pilot courses. The evaluation covered 4 different aspects, all of them strongly positive. Only the "completeness of education" item has a one-fifth share of neutral responses. These results should be used with caution as they represent
Figure 21 and 22: Q10 and Q11 (n = 16, %)
Q10. Was the size of group appropriate?
Q11. Was the level of the learners according to ICT tools appropriate to the course?
Figure 23 and 24: Q12 and Q13 (n = 16, %)
Q12. Was the room equipment Q13. Would you recommend the
appropriate? iFlip training to others?
Neutral 8,3
Positive 41,7
Very
positive
50,0
Probably
yes
40,0
Definitelly 60,0
the educators' opinions on materials developed by them, hence positive bias is very probable.
When asked about the size of the group (Q10), the educators seem to approve it. However, since in the pilot courses the number of participants was not controlled, and participation was freely available, there is little pride in the positive results.
Training organizations should instead be opting for studies into how learners evaluate FTC courses with different number of learners. Similarly, the results for Q11 merely record the status quo without providing insight into what ICT competence level is actually required
for successful learning. We can say that two-thirds of the participants in the pilots had adequate ICT competence level (as judged by their educators), and just over one quarter's competence was considered neutral. Negative responses are 6.5%
At the end of the face-to-face week, after an intensive training on the use of the method and of the ICT tools, the Vietnamese teachers and teams felt more secure on ICT matters, but still expressed the need of a colleague or technician with good ICT skills in their team or their department.
All the teams in both of the Universities were presenting their first flipped classroom course creation, related to their subject and department. They were enthusiastic to use the method, for specific parts of their courses, especially those where students have to repeat or prepare basic knowledge, or where students have to exercise a lot and at several levels.
To make sure teachers who participate at the flipped classroom training start without a lack of self-efficacy concerning their ICT-skills, a face-to-face training on these skills could be organized beforehand. Each team should have one teacher or technician with good ICT skills to support them during the training and during the creation of their own flipped classroom content.
Respondents were almost 92% on the positive side of the evaluation of the training room equipment. Interestingly, and very enthusiastically, all surveyed educators would recommend the iFLIP approach to training to others, with 60% being very resolute about it and 40% saying that they probably will.
Apart from the structured questionnaire, the educators who were trained during the iFlip-project, also made free-text comments on the advantages and disadvantages of the flipped classroom method, which the evaluators have summarized below. The educators who participated at the train-the-trainer training on the method in a flipped classroom setting, with 1 month out-of-class training in advance and 1 week face-to-face training in Ghent, experienced the method themselves while learning about the method. Therefore, we believe they are the main information channels for their colleagues who think of using the method in the future.
Analyzing the answers from the structured interviews and feedback sessions in Vietnam inductively, we can conclude that the participants at the training need a certain level of ICT self-efficacy before the start. This can be realized by training them beforehand, or by providing a colleague or technician with good ICT-skills in their team to support them in learning by doing.
Experiencing the problems with Wi-Fi-connection and availability of software and hardware can be a disincentive.
During the training the teams were trained to formulate objectives, precisely determine which part of their content is most appropriate to
3.2. Conclusion educators
be taught in a flipped classroom setting and how the e-learning materials have to be presented to be attractive for the students and how feedback and evaluations can be organized. The more examples and practical exercises the teachers got during the training, the more creative the teachers became creating the teaching material in a flipped classroom way. The presentation of their materials in between and at the end of the training, led to inspiration and cooperation across the various departments. Each team saw opportunities to successfully integrate the method into their courses in an efficient way, provided that they could devote sufficient time to creating the e-learning materials and did not encounter practical hurdles (Wi-Fi, hardware, software)
3.2.1. Advantages of the Out-of class training
flipped class method The teachers appreciated the out-of-class preparation they had to experienced by the pilot follow before coming to the training, because they could learn at "their group: own speed and learning hours." Important for the pilot group of teachers was also that they could check their progress regularly with a quiz and as such "got good feedback on own progress ".
Face-to-face training
The 5-day-training has been highly valued by the sixteen participants, based on the results from the questionnaire already commented higher on. In the free comments the participants shared some extra considerations, such as "Time spent in class is used more efficiently and goal-oriented" and " iFlip is an innovative approach for teaching".
Designing of own flipped classroom pilot-courses
Once designing their own courses, the teachers indicated that they sometimes struggled with ICT-skills and needed the support of an IT assistant. It was important for them to focus on the course content and didactics of the method and not on the technical implementation, though one of the teachers wrote "that the IT-related work was a great learning experience for me as well". During the designing process, the teachers were becoming more demanding about the quality of the produced media: "The material can easily be developed with free software and basic hardware, but the more you are recognizing and appreciating the strengths, the more you experience the need of higher quality video and sound capture hardware and software".
Testing out the pilot courses with students
During the implementation of the pilots, the teachers learned that the flipped classroom method "provides an ability to reduce time for face-to-face learning and enables differentiation among learners" and that the "individual approach supports active participation of weaker participants" so that "time spent in class is used more efficiently and goal-oriented".
The teachers also appreciated that "the courses/lessons are digital and online and we are able to add resources and activities at any time. The quality of the resources and activities can be improved in the time."
3.2.2. Disadvantages/ pitfalls of the flipped class method experienced by the pilot group
The pitfalls formulated in the free comments can be categorized into technical and organizational issues.
Technical pitfalls for teachers
The fact that ICT skills are needed for both educators and adult learners is outlined as a pitfall "because it requires some technical skills (fluency with programs for creating videos, quizzes, assignments, etc.)". Furthermore "we realize the need of some degree of consistency between teachers' and learners' ICT skills" and "not all learners are familiar with or in possession of ICT devices".
Organizational pitfalls for teachers
The teachers experienced the development of the pilot courses according to FTC methodology as "time consuming". Once testing their pilots, "some learners did not dedicate enough time to view materials in advance and came unprepared to the face-to-face class ".
4. The pedagogical dimension of the flipped classroom method
During the training on the flipped classroom method organized for the iFlip-project, the participants got a technical, didactical and pedagogical training. The purpose was to introduce them to this instructional method, that combines TPACK with the ability of achieving lesson objectives spread over the 3-dimensional framework of Bloom's taxonomy. The training resulted in the designing of pilot courses by the participants. Back at their institutions, these pilot courses were tested out with students (adult learners) and after feedback they were adapted to a final version that could be used throughout the ongoing and future academic years. Analysis of and feedback on these courses, available on http://projectiflip.eu/en/ showed interesting facts on the pedagogical dimension of the method.
4.1. Content analysis
Related to the specialty of the trainers, the courses are all situated in a large variety of subjects: mathematics, physics, language, digital literacy, sociology, pedagogy and science. Before creating their learning paths, we asked the teachers to decide and choose very carefully which and how much of their course content they wanted to teach in a flipped classroom way. Not every kind of content is appropriate for the method, and it is not necessary to transfer the whole course content into a flipped classroom setting.
We concluded that, in terms of content, teachers often choose components of the curriculum, which prepared students to the necessary prior knowledge they needed in order to be able to participate in the actual course. This could be a theoretical or a practical part of the content.
4.2. Audience analysis
Anyone who works with the method as a student must have the necessary ICT knowledge and resources and a good internet connection. But in addition to these purely technical aspects, there is a much more important condition for success. Students must be sufficiently moti-
vated, eager to learn and disciplined to be able to work and study independently. If they come to the face-to-face lessons unprepared, the method will not be of any benefit to them.
4.3. Goal analysis
The method can only be successful if the lesson objectives (spread over the 3-dimensional framework of Bloom's taxonomy) are clearly considered and formulated by the teachers and known by the learners in advance. The media, content, assignments and feedback used must be thoroughly thought through in order to achieve the goals.
The teachers appreciated the fact that they had more strategic class-time: time to teach thoroughly and at a higher level. The learners appreciated that they could prepare the course in advance and learn at their own pace.
4.4. Media analysis
In order to make learning at home as successful as possible, the media provided must be of good image and sound quality. If video is used, the maximum length of the videos may not exceed 3 to 9 minutes, so that the student's attention is not lost. Preferably, the videos and other media are regularly interrupted by questions or tests, for example through interactive questions or a challenge.
4.5. Besign approach
The design of the course is done in an LMS (learning management system). The learning path contains all the media and the lesson content. Thanks to the LMS the teacher can monitor the progress of the students and build in time frames. Feedback and communication is one of the main conditions for success. The LMS provides communication and feedback systems between peers and between the students and the teacher. Using the settings of the LMS, conditions can be built in to be able to move on to the next part of the course, or, if a part is not sufficiently mastered, to build in repetitions or differentiation.
4.6. Organization, methods and strategies of the flipped classroom methods
An evaluation and further studies would be needed to determine whether the flipped classroom method could be used for every subject taught. We recommend that each organization uses a unified pool of resources, interactive tools and structure, so that learners can feel comfortable with new courses and learn in a familiar context and help each other where necessary
5. Discussion Previous research on the flipped classroom method, mostly done to determine the learning outcomes and effects for learners, indicates that "the flipped classroom approach improves the students' learning skills, satisfaction and motivation, without necessary leading to significant better or worse learning performance than in the traditional classroom setting" [Sommer, Ritchhaupt 2018; Flipping First Eras-mus+ Project 2017]. However, in our opinion, the generalizability of
the results is limited by the fact that most of the existing research and evaluation frameworks, for example TAM (Technology Acceptance Model) and HELAM (Hexagonal E-learning Assessment Model), only focus on particular aspects, such as the technological or the learning outcome or the personal effects for the learners but never on all outcomes at whole.
Discussion question 1: How could we design a learners' evaluation framework for the flipped classroom method to evaluate the general, personal and learning outcomes at once, on short and long term?
Discussion question 2: What has the most impact on the learning performance: student engagement, technology and content resources, overall course quality
Discussion question 3: For which target group can the method be most successfully implemented?
The analyzed data on the flipped classroom method before, during and after the iFlip-pilots, reveal that the method is well accepted and highly valued by the teachers, on the condition that they are well trained and prepared both technically and pedagogically before starting to set up their flipped classroom courses and well assisted during the designing process. On these conditions teachers are motivated to put the time and effort into changing their courses [Flipping First Erasmus+ Project 2017; iFLIP Project 2017], fully or partly into a flipped classroom setting. By working together in departments, having access to good practices in a centralized database and having a platform to share know-how, the teachers can support each other as peers [iFLIP Project 2017]. Educators/teachers should be trained and supported in using different tools for course creation. Contemporary ICT skills are needed from both learners and educators.
Discussion question 4: How can we design a teachers' evaluation framework for the flipped classroom method to evaluate the general, personal, course- and teaching outcomes, the pedagogy, the time management, the course quality, the assessment results, the teachers' satisfaction and the teachers' status etc.
Discussion question 5: How are the role of the teacher and the pedagogy of teaching changing with the flipped classroom method?
Discussion question 6: How can the training on the flipped classroom method, realized during the iFlip-training be transferred to a standard training for teachers wanting to use the method
Discussion question 7: What are the minimum requirements to use the method successfully
References Anderson L.W, Kartwohl D. R. (eds) (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Education Objectives. New York: Longman. Bloom B. (ed.) (1956) Taxonomy of Educational Objectives. Boston: Allyn and Bacon.
Darling-Hammond L. et al. (2019). Preparing Teachers for Deeper Learning. Available at: https://learningpolicyinstitute.org/product/preparing-teach-ers-deeper-learning-brief (accessed 15 April 2020).
Dziuban C., Graham C., Moskal P., Norberg A., Sicilia N. (2018) Blended Learning. The New Normal and Emerging Technologies. International Journal of Educational Technology in Higher Education, vol. 15, no 3, pp. 1-16.
Flipping First Erasmus+ Project (2017) Flipclass Handbook. Belgium: Erasmus+.
Fulton K. (2012) Upside Down and Inside Out: Flip Your Classroom to Improve Student Learning. Learning & Leading Technology, vol. 39, no 8, pp. 12-17.
iFLIP Project (2017) Adult Learners Needs Analysis Report. Survey Results and Conclusions. Available at: http://projectiflip.eu/wp-content/up-loads/2018/05/ifLIP IO2 Adult-Learners-needs-analysis-report.pdf (accessed 15 April 2020).
Kwan Lo C., Foon Hew K. (2017) A Critical Review of Flipped Classroom Challenges in K-12 Education: Possible Solutions and Recommendations for Future Research. Research and Practice in Technology Enhanced Learning, no 12, art. no 4. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/ PMC6302872/ (accessed 15 April 2020).
Mishra P. (2019) Considering Contextual Knowledge: The TACK Diagram Gets an Upgrade. Journal of Digital Learning in Teacher Education. April. Available at: https://punyamishra.com/wp-content/uploads/2019/04/TPACK-up-grade-Mishra-2019.pdf (accessed 15 April 2020).
Nurul I., Abdus S., Bhuiyan M., Daud S. B. (2018) A Comparative Study on Achievement of Learning Outcomes through Flipped Classroom and Traditional Lecture Instructions. International Medical Journal, vol. 25, no 5, pp. 314-317.
Rodriguez Moreno J., Agreda Montoro M., Ortiz-Colon A.M. (2019) Changes in Teacher Training within the TPACK Model Framework: A Systematic Review. Sustainability, vol. 11, no 7, art. no 1870.
Sommer M., Ritzhaupt A. (2018) Impact of the Flipped Classroom on Learner Achievement and Satisfaction in an Undergraduate Technology Literacy Course. Journal of Information Technology Education: Research, vol. 17, pp. 159-182.
Talbert R. (2018) What Does Research Say about the Effectiveness of Flipped Learning. Available at: http://rtalbert.org/what-does-the-research-say/ (accessed 15 April 2020).
Teach Tought Staff (2016) A 3-Dimensional Model Of Bloom's Taxonomy. Available at: https://www.teachthought.com/critical-thinking/3-dimensional-mod-el-blooms-taxonomy/ (accessed 15 April 2020).
Tomas L., Evans N., Doyle T., Skamp K. (2019) Are First Year Students Ready for a Flipped Classroom? A Case for a Flipped Learning Continuum. International Journal of Educational Technology in Higher Education, vol. 16, art. no 5.
