CC BY
38
CONTINUOUS QUALITY IMPROVEMENT (CQI) FRAMEWORK: A CASE OF INDUSTRIAL ENGINEERING
DEPARTMENT
Tooba Sikander, Research Scholar, Department of Engineering Management, University of Engineering and Technology, Taxila, Pakistan, E-mail: toobasikander@yahoo.in Dr. Haris Aziz, Industrial Engineering Department, University of Engineering and Technology Taxila, Pakistan,
E-mail: haris.aziz@uettaxila.edu.pk Dr. Ahmad Wasim, Industrial Engineering Department, University of Engineering and Technology Taxila,
Pakistan, E-mail: wasim.ahmad@uettaxila.edu.pk Dr. Salman Hussain, Industrial Engineering Department, University of Engineering and Technology Taxila,
Pakistan, E-mail: salman.hussain@uettaxila.edu.pk Dr. Mirza Jahanzaib, University of Engineering and Technology Taxila, Pakistan E-mail: jahan.zaib@uettaxila.edu.pk
A R T I C L E I N F O
Original Research Received: October, 31.2016. Revised: December, 21.2016. Accepted: January, 01.2017. doi:10.5937/IJCRSEE1701107S UDK
378:006.83(549)
Keywords:
Outcomes based education, continuous quality improvement framework, Bloom's Taxonomy, Outcomes based accreditation, program learning outcomes, course learning outcomes.
A B S T R A C T
This paper aims to present an educational framework for outcomes based continuous quality improvement. Well defined program outcomes, program educational objectives and assessment process have been developed to ensure graduates' outcomes achievement. Direct and indirect tools have been used for assessment process. Course evaluation surveys, alumni surveys, and employer surveys have been deployed for indirect outcome assessment. Exams, quizzes, assignments and projects, on the other hand, have been used for direct outcome assessment. In developed framework, the educational processes committees and facilities committees have been integrated to continuously evaluate and monitor the educational processes. Furthermore, program outcomes and course learning outcomes are proposed to be evaluated and continuously monitored by programs goals committee and continuous course improvement committee respectively. Forms and procedures have been developed to assess student outcomes.
© 2017 IJCRSEE. All rights reserved.
1. INTRODUCTION
International engineering alliance (IEA) gave a signatory status of Washington Accord (WA) to Pakistan Engineering Council in 2010. (Manual of accrediation Engineering Accrediation Board, 2014). Before 2002, the Washington Accord was based on conventional education system widely known as input based education system. The major problem associated with conventional education system was non-provision of feedback; which was major hindrance in continuous quality Corresponding Author
Dr. Mirza Jahanzaib, University of Engineering and Technology Taxila, Pakistan,
E-mail: jahan.zaib@uettaxila.edu.pk @©@©
This work is licensed under a Creative Commons Attribution - NonCommercial - NoDerivs 4.0. The article is published with Open Access at www.ijcrsee.com
improvement (Patil and Codner, 2007). After the identification of drawbacks in conventional education system, Washington Accord adopted outcome based education system in 2002 (Laguador and Dotong, 2014). After obtaining signatory status in 2002, Pakistan Education Commission faced the major challenges of conversion of their education system from conventional to outcome based system.
To overcome this issue, there was a need of continuous quality improvement framework for outcome based education implementation. The framework was required to integrate course objective, curriculum mapping, direct/ indirect measure and continuous course quality improvement (CCQI) as a single entity.
The paper aims to propose a continuous quality improvement framework for engineering education in Pakistan. The paper is structured as follows. Literature review has been presented in Section 2. Section 3 describes the methodology adopted for the development
of framework. The proposed framework has been provided in Section 4.
Section 5 presents the results and analysis obtained by the application of proposed framework for a specific course. Finally Section 5 and 6 present conclusions and recommendations respectively.
2. LITERATURE REVIEW
Outcome based education is a newly developed model for the improvement of quality education. (Rajaee et al., 2013). It focuses on teaching and learning based on desired student's learning outcomes (Spady, 1994). To achieve these outcomes, assessment and performance standards have been designed and implemented (Spady, 1994). Literature review highlights the significance of outcome based education (Borsoto et al., 2014). Furthermore it serves as competitive advantage among different countries (Spady, Marshall and Kit, 1991).
In past centuries, two major education systems have been proposed namely (i) problem based learning, and (ii) project based learning. Problem based learning is a student centered learning approach. It encourages teachers to be facilitators instead of disseminators. It is based on open ended problems which serve as initial stimulus (White, 2001). Project based learning, on the other hand, aims to engage students in authenticated problems investigation. It is based on project with the objective to provide learning (Blumenfeld et al., 1991). Both traditional educational systems focus on contents with predefined curriculum and assessment system. Outcomes based education system, on the other hand, is relatively a new system which is based on defined framework for outcomes (Spady, 1994) i.e. it focuses on student's outcomes and skills they must possess in future (Spady, Marshall and Kit, 1991). In this system, curriculum, assessments, instructional strategies and performance standards are developed and implemented to meet desired outcomes (Spady, 1994).
Outcome based education includes (i) program educational objectives (PEO's), (ii) program learning outcomes (PLO's), (iii) stakeholders involvement in defining PEO's and PLO's, a well curriculum development to achieve desired program educational objectives and program learning outcomes, (iv) courses mapping to program educational objectives and program learning outcomes for
contribution and development of continuous improvement process (Rashid, 2012).
Program educational objectives describe career and professional success that students attain after graduation (Manual of accredia-tion Engineering Accrediation Board, 2014). It is classified in to six categories namely (i) knowledge, (ii) comprehension, (iii) application, (iv) analysis, (v) synthesis and (vi) evaluation (Student Outcomes and Program Educational Objectives, 2015). Program learning outcomes have to be attained by students at the time of degree completion and must be specific, measureable, achievable, realistic, and time-based (Kapfer, 1971).
Assessment is the third step which aims to defining clear and measureable outcomes, ensuring opportunities to achieve these outcomes, implementing an evaluation system and then use results from evaluation for student learning improvement (Bresciani, 2006). Many direct and indirect assessment tools have been proposed in literature. Direct assessment tools include course assessment, student satisfaction survey, cumulative GPA index for course, basic exams, senior design project, and program accreditation (Sekhar et al., 2008). Senior exit survey is an indirect program outcomes assessment tool (Oth-man et al., 2011). Teacher evaluation is a tool that can be used for the evaluation of teacher performance (Chalmers and Gardiner, 2015) by student feedback which helps to improve teaching skills. Teacher training programs are also designed on the basis of evaluation which aid to enhance teachers' performance (Boerbooma et al., 2015). Rubrics is another indirect assessment tool that can be used for student's skills and course assessment (Al-marshoud, 2011). Core indicators affecting academic institution are prioritized in three layers. Leadership, quality of faculty and infrastructure facilities reside at first layer. Quality of students, research, and learning environment are in second layer followed by strategic planning, governance, assessment procedures and market force as the third layer (Jahanzaib and Akhtar, 2005).
Review of literature presents that there exists a number of outcome based education models and continuous improvement frameworks. Bloom and his colleague Maser worked on behavioral objectives and developed taxonomies of the objectives for cognitive and effective domain (King and Evans, 1991). Davis adopted Carroll and Bloom's work to achieve desired educational reform (Davis, 2003). The developed framework con-
sists of three phases. First phase of framework involves program and course objectives development; Second phase includes defining student learning outcomes based on Bloom's taxonomy of cognitive domain followed by third phase involving assessment workshops to familiarize faculty and administration with assessment techniques (Mcgourty et al., 2002). Besterfield proposed a conceptual model for quality improvement in engineering education and developed program outcomes using affinity diagrams (Besterfield-sacre, Shuman, and Wolfe, 2002). Volkwein, Lattuca and TereN-zini developed a conceptual framework to examine impact of ABET's EC200 impact on student's learning outcomes (Volkwein, Lattuca, and TereNzini, 2004). Malaysian institute of information and technology developed a computerized system to link, asses and measure course learning outcomes and program learning outcomes (Mokhtar et al., 2014). Beside these typical assessment models a new program outcomes assessment model focusing on performance criteria for each program outcome. The researcher discouraged the use of grades as performance indicators (Mohammad and Zaharim, 2012). A continuous improvement program based on outcomes based education comprised of student's outcomes assessment tools have been developed and implemented by Sekhar, C. R. et all. (2008). An outcome driven hybrid academic model was proposed by Bouslama et al (2003). It uses grade point average along with outcomes driven students output. The model consists of learning outcomes, use of e-portfolios to record student achievement, assessment and its reflection and closed feedback learning communities.
From the detailed literature review, it can be seen that number of authors have proposed outcome based education implementation models. However, it must be noted that systematic implementation of framework is required to fully achieve objectives of outcome based education. Engineering education institutions in Pakistan are bound to follow outcome based education in true spirit to get accreditation from Pakistan engineering commission. Therefore this research has proposed outcome based education continuous quality improvement framework. This will help the engineering institutions in Pakistan to apply outcome based education in true manners.
3. METHODOLOGY
This research explores the continuous quality improvement framework for outcome based education implementation in engineering institutions of Pakistan. The methodology consists of five phases. Literature review has been conducted in first phase to identify the limitations of previously developed frameworks. After identifying the gaps in phase 1, a framework has been proposed in phase 2. The framework aligns the classroom practices and educational structure. A specific course was selected in phase 3 for the implementation of framework. Analysis of the course assessment data was conducted during phase 4. Finally, improvement methodology was suggested in phase 5 for continuous quality enhance.
4. PROPOSED FRAMEWORK
An outcomes assessment and continuous improvement framework based on above mentioned methods are proposed to ensure program educational achievement. The proposed framework has been presented in figure 1.
The proposed framework consists of three modules namely (i) Outcome module,
(ii) management module and (iii) assessment module. Outcome module defines the targeted outcomes and it consists of program educational outcomes, program earning outcomes and course learning outcomes. Assessment module assesses the fulfillment of the targeted outcomes based on certain criteria. Management module consists of monitoring and evaluation committees for continuous improvement. Three suggested committees are: (i) Continuous course improvement committee (CCI), (ii) program goals committee (PGC) and (iii) education facilities and processes committee (EFPC).
Significant feature of proposed framework is indicated by three layers of outcomes assessment cycles.. These cycles includes (i) course learning outcomes assessment, evaluation and continuous improvement cycle, (ii) program learning outcomes assessment, evaluation and continuous improvement cycle and
(iii) program educational objectives assessment, evaluation and continuous improvement cycle.
First cycle encloses course learning outcomes assessment and improvement cycle. Closing the loop takes sixteen weeks (one semester). The second cycle program outcomes
assessment cycle is a slow process and takes one year. Third cycle is program educational objectives assessment and improvement which is also slow in nature. This cycle take
four years to close it. Stakeholders (students, alumni, industry, parents, and faculty) provide input in form of surveys, gathered data is analyzed to review and modify PEO's and PLO's.
Outcome Module
Program Educational
objectives
Educational Proem
Program Learning Ou lecMTif >
Course Learning Outcome*
Program g culi commit!*»
Conttniwut
cours« iTprovament commlttse
Educator faculties and praciimua
tornmU'.M
L
Management Module
Assessment Module
DIR ECT INDIRECT
Program Educational objective* /
* V* •i Program Learning Qu ¡cornos * •i
* Course Learning Outcome* v"
Exams Quizzes Assignments Cours* Evaluation Survey SiudMi Exit Survey Alumni. Industrial Survey
Figure 1. Outcomes Based Education Continuous Quality Improvement framework
Outcomes Assessment methods used to evaluate outcomes according to developed framework are direct assessment and indirect assessments. Direct assessment includes outcomes assessment (e.g. course learning and program learning outcomes). Indirect assessment, on the other hand includes student's course completion survey, exit survey, alumni survey and employer survey
These assessment processes may be used to improve outcomes, course contents, curriculum and educational processes to achieve targeted outcomes. Teacher evaluation survey may be used to evaluate teacher performance.
Following sections explains the implementation of the proposed framework for course assessment and continuous improvement. The implementation starts with setting target outcomes (PEO, PLO and CLO), PLO/ CLO mapping, assessment and feedback for improving outcomes. All procedures been discussed separately below.
4.1. Setting target outcomes
This step involves setting target program educational outcomes, program learning outcomes and course learning. Program learning outcomes are statements which detail skills and abilities undergraduate students must achieve at time of degree completion. Program educational objectives and program learning outcomes have been developed as per Pakistan engineering council's guidance and are presented in table 1. Each course in the entire industrial Engineering curriculum contributes to achieve some or many of the program learning outcomes. Correlation of
current Industrial engineering courses with program learning outcomes has been presented in Annexure 1. For more detailed analysis, correlation of program learning outcomes of one of the selected course "Metal forming and cutting analysis" is shown in table 2.
Table 1. Program educational objectives and program learning outcomes
a_ To produce Industrial Enginero vvto assume challenging managerial and lEadership roles mhtmulac-turins arid Service sectors both at Xationai axj d inteituii:]cjual Lev tL
b. To provide Employers Tvith graduates ivko remain it forefront and are teclroicaJh" competent; tavE inter-persiina;. decision mak_iig ana problem solving skills bo enhance effectivieness
c. To produce professionals who recognize that EHEineeriuE is a global service profession that c:ust be
EjiEuieerLiE Kao*vl&dae: An ability to app]y bnioivledge of nathEmatica. Lcisnne and eugicEaring to
b. Problem Analysis: An ability to identify. formulate. Eolve and analyse enEineerinE problems.
c Design-Development ot t.Dlutein An ability to design a system. component. or process to meet desired needs withinrealistic constraints ¿uit ae esonomic. envTiornijeniai. icciai. poj:-iial. ethicaL health ar.d
d. Investigation: An ability to investigate complex engineering problaaa hjstematically Euch as literature review, cesien and conduct e^pomieiils, analysis and! interpretation of experiments to derijre valid sm-
e. Modem ToolL'sage: An ability to create, select and apply appropriate techniques. slcils, and modem enEineerinE tools necessary for ELEineerins practice
£ Tiie Engineer and Society-: An ability (d apply reasoning informed by conteirtual knowledge to assess societal, health, safely, lesal and cuitujal i^ufe and the consequent responsibilities relevant to professional engineering practice and solutionis to complex engineering problems
e. Environment and Suztainability: An ability to understand the impact of engineering iohidons u: a Eiobal, economical. societal uonJtEKL environmental and E'lstainable development
h. Ettncs: Ail ability" № understand Ethical and professional responsibilities
i. Individual auii Tears Work: An ability to vvotk effectiv ely as En individual or in a team
k. Lifelong I earning: An ability to recognize the nesd far innovation and technological development and to
I. Project Mamganat An ability to apply concepts of industrial system and process desian. ergonomics. enEineerinE management, product development 'manufadmng and analysis tools to wk in a team effectively and to manage projects in miiiti disciplinary EJLvirotirr.eiLts
Table 2. Program learning outcomes for Metal forming and cutting analysis PLOs_PLOs Statements_
PLO 1 :a ail ability to apply knowledge of mathematics, science and engineering to solve complex problems
PL02:b an ability to identify, formulate, solve and analyze engineering problems
PL03 :c an ability to design a system, component, or process to meet desired needs within realistic constraints such, as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustains bility
PL04:e an ability to create, select and apply appropriate techniques, skills, and modern engineering tools necessary for engineering practice
A set of measureable course learning outcomes are required to drive the course. A sample of course learning outcomes developed for the course "Metal forming and cutting analysis" are shown in table 3. CLO's are developed on account of skills students must possess by the course completion. Course
experts took sixteen weeks to deliver course contents assisting achieve course learning outcomes.
Table 3. Course learning outcomes for Metal forming and cutting analysis
Recalling the concepts of Manufacturing Processes for in-depth knowledge of processes for enhanced learning. Under
Identification of the known parameters in ^ forming and cutting processes and making efficient use of boundary conditions AP-
Be able to apply related conditions (of parameters) and obtaining an appropriate solution within given constraints EVAL-
Reinforce evaluation capability for further synthesis of problems in metal forming cutting analysis for creation of prod-
4.2. Correlation between Program learning outcome and course learning outcome
Correlation assessment between course and program learning outcome allow program to know course contribution to outcomes but it do not allow assessment of specific outcomes achievement. For its assessment program learning outcomes are divided into number of measureable statements to identify student's performance required to meet outcomes. These measureable statements are called performance criteria. For example PLO j "an ability to communicate effectively" would require criteria i.e. (i) effective presentation (ii) effective writing (iii) professional communication mastery. These three criteria require three types of assessments and measurement.
Graduate must achieve all program outcomes by the end of degree completion which calls for a procedure to measure program learning outcomes by course completion. To measure PLO's course learning outcomes are mapped to program learning outcomes by a linking matrix. In this way PLO's are measured by accumulated sum of CLO's. For continuous improvement course learning level monitoring is required to make sure contents delivered and assessed are in accordance with course com-
plexity level or vice versa so, course learning outcomes (CLO's) are mapped with Bloom's taxonomy. In addition, course contents, teaching methods and assessment methods need to be documented and continuously monitored. Student satisfaction surveys conducted after course completion assist teaching learning methods, assessment methods and course contents continuous improvement where as a documentation procedure shown in table 4 is developed which helps reviewing teaching learning methods, course contents and assessment methods in case of any CLO failure.
Table 4. Curriculum mapping
Learning Teaching Assessment
CLO's PLO's Course contents
Level methods method
Under- Introduction to MFCA, joining processes Lecture, Quiz, Home
la
stand and wear theory presentations, work
Orthogonal Cutting Model in Machining.
Lectures,
Merchant Equation, Surface Finish and cut- . Quiz, Home
b Apply _ presentation,
ting Tools Technology, Plastics extrusion, work
discussion
sheet metal cutting
Forces calculations in metal cutting for
Quiz, Home-
evaluation, Forging analysis for pressure Presentations, 3 c Evaluate work,
evaluation in billets, sheet metal cutting, discussion
Exam
grinding or machining process evaluation
4 e Create Mini Project/Capstone project Discussion Presentation
Under a course offered by a degree program course learning outcomes are required to be directly and indirectly assessed and measured. Table 5 shows procedure assisting outcomes measurement process. Assessments are the instruments used by instructor to assess student's skills. One assessment method used for assessing a CLO is suggested but in present case instructor used more than one method to assess single CLO's. Table 5 shows a matrix linking assessment questions to CLO's to measure each CLO separately in addition with performance criteria for each assessment.
Table 5. Mapping assessments with CLO's
Assessment techniques question CLO Performance criteria
Home work IIW 1 5 Qs 2 an ability to apply knowledge of mathematics, science and engineering to solve complex problems
HW 2 5Qs 2 an ability to identify, formulate, solve and analyze engineering problems
HW 3 5 Qs 2,3 an ability to create, select and apply appropriate techniques, skills, and modern engineering tools necessary for engineering practice
Quiz 1 2 Qs 1=2 an ability to apply knowledge of mathematics, science and engineering to solve complex problems
Quiz Quiz 2 2 Qs 2 an ability* to identify, formulate, solve and analyze engineering problems
Quiz 3 2 Qs 3 an ability to create, select and apply appropriate techniques, skills, and modern engineering tools necessary for engineering practice
Q1 — —
Q2 2
Mid- Q3 an ability to identify, formulate, solve and analyze engineering problems
Exam term Q4 3 an ability to create, select and apply appropriate techniques, skills, and
exam modern engineering tools
Q5 3 necessary for engineering practice
Q6
Any Mini other Project
ability to create, select and apply appropriate techniques, skills, and modem engineering tools necessary for engineering practice ability to apply reasoning informed by contextual knowledge to assess societal, health, safety, legal and cultural issues and the consequent responsibilities relevant to professional engineering practice and solutions to complex engineering problems
4.3. Outcome assessment
Course instructors must decide to setup instruments to assess course learning outcomes in their course. In present case Student's skills and abilities are directly assessed by homework's, quizzes, exams and projects.
Indirect assessments are student's perception based assessments on their outcomes achievement and understanding. Program learning outcomes (PLO's)and course learning outcomes (CLO's) are indirectly assessed by student's satisfaction survey conducted after course completion. Student satisfaction survey was composed of two parts;1) Achieving of learning outcomes 2) course management. For this purpose likert scale (1-5) was used and students were asked to rate understanding of each CLO and PLO and course management which is described later.
To evaluate student's outcomes a threshold indicator is required to make sure achievement of outcomes up to a certain level. This is to mention that student needs to obtain 60
percent marks to achieve desired program and course learning outcomes while 60 percent students must achieve course and program learning outcomes for a course to be successful. To measure particular course learning outcomes for a student 'marks obtained by student are tabulated in excel form and then converted to percentage form. Then successful student's percentage for particular CLO under the course is calculated to find achievement. Achievement indicators for outcomes are as follows:
90-100 % fully achieved 80-90 % excellently achieved 70-80 % achieved 60-70 % satisfactoilry achieved Below 60 % failed In case of course learning outcome (CLO's) correlation with single program learning outcome (PLO's) its data is directly used for PLO measurement but in multiple CLO's correlation with single PLO case, weight must be defined for each CLO i.e. Percentage contribution of CLO to achieve corresponding
PLO and final marks are composite sum of individual CLO's at specific percentage and normalized to 100 percent. PLO achievement percentage is then measured by accumulating relevant CLO's data according to defined weights. Present case is an example of single CLO's correlation with PLO. Mathematical model for PLO measurement is given below Step 1: V i determine percentage scores of students
Step 2: V i compute the average percentage of scores to get aco; average points Step 3: V i assign percentage weightages w to each j
Step 4: Compute the sum of weightages by
Step 5: Compute the sum of Ih^ACO, . v j Step 6: Get the PO contribution by
5. RESULTS AND ANALYSIS
Figure 2 shows outcomes achievement for "Metal forming and cutting analysis" course. As course learning outcomes (CLO's) mapping with program learning outcomes (PLO's) shows one to one correspondence so the graph shows results for both PLO's and CLO's. It can be seen from figure 2 that successful student's percentage is greater than 60% for all PLO's so the course is successfully completed with achievement of all CLO's
and PLO's.
f
CLO's Achivement
Lischiem) not achieved
V_J
Figure 2. Course Learning Outcomes attainment
5.1. Direct and indirect assessments result comparison
Direct and indirect assessments need
same basis of performance indicators to allow comparison. So Direct assessments percentage marks were converted to indirect assessment scale. For this purpose students percentage marks were divided by 20 to convert in to a linier scale i.e. 1 - 5. Performance indicators for assessments are set as: For an outcome to be successful both direct and indirect assessment calculated value must be greater than three. Direct and indirect assessment results agree with indirect assessment results as given in table 6 and 7 which show students actual learning and understanding. It is evident to say that direct and indirect assessments are the instruments to assess student's outcomes.
Table 6. CLO's direct and indirect assessment comparison
Direct assessment Indirect Assessment Remarks CLO Successful Sr.ill- 5 Mean Standard students % deviation
1 63 3.1 4.3 0.8 Successful
2 75 3.7 4.3 0.6 Successful
3 91 4.5 4.3 06 Successful
4 75 3.7 4.1 0.6 Successful
Table 7. PLO's direct and indirect assessment comparison
Direct assessment Indirect assessment Remarks
Successful students % Scale=5 Menu Staudard deviation
63 3.1 4.2 0.7 Successful
75 3.7 4.3 0.7 Successful
91 4.5 4.0 0.9 Successful
75 3.7 4.0 0.9 Successful
5.2. Measuring Course level by using Bloom's taxonomy
Engineering courses are being taught in an order of learning level and complexity i.e. higher learning level course is offered in higher semester. Continuous quality improvement calls a procedure for course learning level assessment for each course offered. To ensure the achievement of acceptable course level by an 6instructor, selected course learning level was examined by mapping assessment questions to Blooms taxonomy. The frequencies of cognitive levels of assessment questions are given in table 8 for the specific course. It can be seen that 52% questions used for assessments were at understanding level, 8.6 % at application level and analyzing level, 21 % at evaluation level and only 8.6% at synthesis level. Maximum marks percentage of each cognitive level
was evaluated. It can be established from table 8 that the course "metal forming and cutting analysis" was taught at 5th level of learning.
Table 8. Assessing course learning level
Questions Level (Blooms) Questions frequency Percentage Max marks Percentage
Knowledge 0 0 0 0
understanding 12 5217 5.7Î 19.16
applying 2 8.6 1 3.3
analyzing 2 8.6 S 42.6
évaluation 5 21.1 12.25 40.8
synthesis 2 8.6 8 26.6
total 23 100 30 100
5.3. Course evaluation
Review of literature resulted in factors affecting student's performance as shown in table 9 which are categorized into two parts: (1) outcomes and (2) course management. Former assesses course and program learning outcomes while later assess course contents, teaching learning methods and assessment methods. To assess course management Lik-ert scale 1-5 has been used. Table 9 details the means scores and standard deviation obtained in result of course evaluation survey analysis after course completion.
Table 9. Course Evaluation survey
Factors affectiag student performance MeaD SD
Course * Clearly expressed 4.0 0.7
objective
PLO * Attainment 3.8 0.8
* Strengthened course un-
Course demanding 4.0 0.8
contents * Helped achieving course objectives
* Comprehensive
Teaching * Enhanced skills and abil- 4.1 0.9
learning ities relevant to course
methods
* assessed CLO s
* assessed Strength and 3.9 0.9
Assess- weakness
ments * helped Learning * Level was appropriate to given credit,
CLO * Attainment assessment 4.02 0.8
5.4. Teacher Evaluation
Review of literature resulted in multiple teacher traits resulting positive students learning outcomes which can be stated as maintaining positive learning climate, class instructions, class management, and student involvement in class, teaching/learning and student differentiation. Table 10 details mean scores and standard deviation obtained in result of teacher evaluation after course completion.
Table 10. Teacher Evaluation survey
Teachers traits affecting students performance Mean SD
* Student's respect
Learning Climate * * Relaxed learning environment Support for Class participation Student collaboration 4,6 0.6
« Declared lesson objective
Class instruction * Well structured lesson Communication skills Students comprehension 4.2 0.8
* Lecture's start and finish time
Class management * Appropriate Students progress evaluation techniques 4,2 0.7
* Class room rules
* Critical thinking encouragement by ask-
Student involvement in class ing questions * Student's interest in class 4.1 0.7
* Students competition
* Encourage students to think on solutions
Teaching as learning * * Apply learning practically Problem solving strategies Challenging assignments 4.1 0.8
Student differentiation * Boost weak student' s self confidence Additional instructions time 3,3 1.3
6. CONCLUSIONS
Due to requirement imposed by Washington accord agreement, engineering education in Pakistan is in the phase of transformation from traditional education to object based education system. Program learning outcomes assessment and evaluation is mandatory now for all engineering programs in Pakistan. In order to ensure that the graduates of industrial engineering department have achieved program educational objectives, an outcomes based accreditation continuous improvement framework has been presented. The assessment of engineering courses with respect to learning outcomes and student's expectations is an important activity regarding course improvement. Therefore course learning outcomes were directly and indirectly assessed. It was found that the proposed method has the ability to gauge achievement of student's outcomes. Strengths of the model include; (i)
clearly defined performance criteria which are evidence for specific program educational objective, (ii) performance criteria is linked with evidence, (iii) use of performance indicators instead of grades.
Continuous quality improvement can be implemented based on these strengths. Proposed method is a close loop model equipped with continuous quality improvement.
7. SUGGESTIONS AND RECOMMENDATIONS
The findings of this research include:
• The developed framework facilitates continuous quality improvement while developed forms and procedures assess students' outcomes.
• Student exit, alumni and employer surveys are tools to evaluate outcomes at program level while course evaluation survey is an instrument to evaluate outcomes at course level.
• Student exit, teacher evaluation, and course evaluation survey are instruments to improve educational processes assisting attain educational objectives
• Program goals committee evaluates and monitors outcomes at program level while educational processes and facilities committee evaluate and monitor program performance by conducting surveys. Continuous course quality improvement committee is recommended to overcome course weaknesses.
Procedure to improve course followed by Program outcomes assessment tools are described below
7.1. Continuous course quality improvement
The results analysis section showed both direct and indirect measures for the course selected. It is evident from direct and indirect measures that course is successful but some percentage of students failed in program learning outcome "a", "b", "c" and "e". There is need for continuous course quality improvement monitored by CCQI committee to minimize failure percentage for same course next time. Given below methods are recommended/ suggested to figure out failure reasons.
7.1.1. Failure patterns
To address this issue continuous course improvement committee (CCQI) committee systematically must use set of information related to course and its prerequisite's course:
(1) outcomes correlation with the prerequisite and present course
(2) Students failure percentage in prerequisite
(3) switch from very low learning level (for pre-requisite) to very high learning level (present course)
(4) pre-requisite Course contents
(5) Level assessed in pre-requisite course exams.
7.1.2. Issues related to student failure
Survey may help identify issues related to student failure i.e. poor course management or student's personal problems. In case of poor course management actions must be taken to improve course.
7.2. Program Learning Outcomes (PLO's) and Program Educational Objectives (PEO's) assessment tools
Program learning outcomes and program educational objectives need to be continuously assessed and monitored. Several tools have been developed to indirectly assess student's outcomes at the time and after degree completion. Following section describe complete details of tools used for indirect assessment.
7.2.1. Student Exit survey
Review of literature resulted in number of factors affecting academic institutions performance and highlighted significance of student exit survey to evaluate institution performance and assess student's outcomes. Factors affecting academic performance can be stated as: (1) Education quality (2) Facilities and (3) Curriculum. Student exit form has been developed and categorized in to two sections: (1) Assessment of outcomes and (2) Academic performance evaluation. Table 10 show all factors enclosed in student exit survey in detail.
7.2.2. Alumni survey
ACKNOWLEDGEMENTS
Alumni form also consists of two sections: (1) PEO's assessment and (2) alumni career. First section assesses outcomes while second section deals with graduate satisfaction with career, employment status, confidence in work and commitment to lifelong learning. Table 10 show all factors enclosed in Alumni survey in detail.
7.2.3. Employer survey
The authors would like to record their appreciation to the Head of Industrial Engineering Department, University of Engineering and Technology Taxila (Pakistan) for funding this research. They also wish to thank all the participants who responded timely and provided their feedback.
Conflict of interests
The authors declare no conflict of inter-
est.
Employer form has been developed to collect employer's views about alumni performance, employed at industries in addition with industry demands i.e. which skills employers demand for an engineer to be employed at their organizations. Program educational objectives and program learning outcomes assessment in addition with modification to compete globally is purpose of employer form as shown in table 11.
Table 11. Student exit, Alumni and Employer survey
Knowledge Skills ¡ attributes
Education quality
Facilities
General opinions
program educational objectives
Satisfaction with career
Graduate employment statuses
Confidence in work
Commitment to lifelong learning
Knowledge Skills and attributes
Program educational objectives
General assessment
clearly declared Attainment assessment Modify /delete/add
Instructions quality and knowledge Professional practices development
Computational facilities Laboratory facilities
Equipment needed for a modem education Library7 holdings_
Team projects and discussions Missing material Unnecessary material Career development_
Important skills learned during graduation
Meet newly admitted students Modify/ delete / add
Current employment position Successful Career
Find job within 1 year after graduation
Job is closely related to education Skills fulfill job demand Engineering courses that need improvements
Missing material
professional society Enrolled in education
Academic and professional preparation Industrial skills demand Modify/delete.1' add
Role in global Business Attainment assessment Mo dify/delete/add
Suggestions to improve
REFERENCES
Almarshoud, A. F. (2011). Developing a rubric-based framework for measuring the ABET outcomes achieved by students of electric machinery courses. International Journal of Engineering Education, 27(4), 859. http://www.ijee.ie/con-tents/c270411.html Besterfield-Sacre, M., Shuman, L. J., & Wolfe, H. (2002). Modeling undergraduate engineering outcomes. International Journal of Engineering Education, 18(2), 128-139. http://www.ijee.ie/ articles/Vol18-2/IJEE1269.pdf . Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting the learning. Educational psychologist, 26(3-4), 369-398. http://dx.doi.org/10 .1080/00461520.1991.9653139 Boerboom, T. B., Stalmeijer, R. E., Dolmans, D. H., & Jaarsma, D. A. (2015). How feedback can foster professional growth of teachers in the clinical workplace: A review of the literature. Studies in Educational Evaluation, 46, 47-52. http:// dx.doi.org/10.1016/j.stueduc.2015.02.001 Borsoto, L. D., Lescano, J. D., Maquimot, N. I., San-torce, M. J. N., Simbulan, A. F., & Pagcaliwa-gan, A. M. (2014). Status Of Implementation And Usefulness Of Outcomes-Based Education In The Engineering Department Of An Asian University. Asian Journal of Management Science and Economics, 1(1). http://www. multidisciplinaryjournals.com/wp-content/up-loads/2014/02/STATUS-0F-IMPLEMENTA-TION.pdf
Bouslama, F., Lansari, A., Al-Rawi, A., & Abonamah, A. A. (2003). A novel outcome-based educational model and its effect on student learning, curriculum development, and assessment. Journal of Information Technology Education, 2, 203-214. http://www.jite.informingscience.org/ documents/Vol2/v2p203-214-22.pdf Bresciani, M. J. (2006). Outcomes-based academic and co-curricular program review: A compilation of institutional good practices. Stylus Publishing, LLC. (ISBN: 1-57922-140-8). Chalmers, D., & Gardiner, D. (2015). An evaluation framework for identifying the effectiveness and impact of academic teacher development programmes. Studies in Educational Evaluation, 46, 81-91. http://dx.doi.org/10.1016Zj.stue-duc.2015.02.002
Davis, M. H. (2003). Outcome-based education. Journal of veterinary medical education, 30(3), 258263. http://dx.doi.org/10.3138/jvme.30.3.258 Jahanzaib, M., & Akhtar, K. (2005). Prioritizing key quality indicators for engineering education using analytical hierarchical process (AHP): Case study. (pp. 1-12). In Proceedings of Pakistan's 4th conference on quality assurance in education, organized by Pakistan Institute of Quality Control, Lahore on (pp. 17-18). Kapfer, M. B. (1971). Behavioral objectives in curriculum development: selected readings and bibliography. Educational Technology. (ISBN: 0-87778-001-3) King, J. A., & Evans, K. M. (1991). Can we achieve outcome-based education. Educational Leadership, 49(2), 73-75. http://ascd.com/ASCD/pdf/ journals/ed_lead/el_199110_king.pdf Laguador, J. M., & Dotong, C. I. (2014). Knowledge versus Practice on the Outcomes-Based Education Implementation of the Engineering Faculty Members in LPU. International Journal of Academic Research in Progressive Education and Development, 3(1), 63-74. http://dx.doi.org/ 10.6007/IJARPED/v3-i1/640 Learning, P. B. (2001). Speaking of Teaching. (Winter 2001, 11(1), 1-6. Stanford University Newsletter on Teaching. http://www.konstruktivismus. uni-koeln.de/didaktik/problembased/problem_ based_learning.pdf Majid, F. A. (2008). The use of reflective journals in outcome-based education during the teaching practicum. Malaysian Journal of ELTResearch, 4. http://s3.amazonaws.com/academia.edu.do-cu-ments/13774581/faizah_v2.pdf?AWSAcces sKeyId=AKIAJ56TQJRTWSMTNPEA&Ex-pi res=1483958766&Signature=O6K6z8lc2P8R L1ggm6HlE6i9tcE%3D&response-con-tent-disposition=inline%3B%20filename%3DThe_ Use_of_Reflective_Journals_in_Outcom.pdf Manual of accrediation Engineering Accrediation Board. (2014). Retrieved may 20, 2015, from http://www.pec.org.pk/downloadables/Accredi-tation/PEC%200BA-Manual%202014.pdf McGourty, J., Shuman, L., Besterfield-Sacre, M., At-man, C., Miller, R., Olds, B., ... & Wolfe, H. (2002). Preparing for ABET EC 2000: Research-based assessment methods and processes. International Journal of Engineering Education, 18(2), 157-167. http://drjj.uitm.edu.my/DRJJ/ MQAGGPAS-Apr2011/ABET-LO-CRITERIA/ ABET-Assessment-IJEE1278.pdf Mohammad, A. W., & Zaharim, A. (2012). Programme outcomes assessment models in engineering faculties. Asian Social Science, 8(16), 115. http:// dx.doi.org/10.5539/ass.v8n16p115 Mokhtar, S. A., Puteh, S., & Anuar, S. M. S. (2014). OBE Measurement System in Malaysian Institute of Information Technology Universiti Kuala Lumpur. In 2014 5th International Conference on Intelligent Systems, Modelling and Simulation (pp. 12-17). IEEE. http://dx.doi. org/10.1109/ISMS.2014.10 Othman, N. T. A., Misnon, R., Abdullah, S. R. S., Kofli, N. T., Kamarudin, S. K., & Mohamad, A. B. (2011). Assessment of programme outcomes through exit survey of Chemical/Biochemical Engineering students. Procedia-Social and Behavioral Sciences, 18, 39-48. http://dx.doi. org/10.1016/j.sbspro.2011.05.007 .
Patil, A., & Codner, G. (2007). Accreditation of engineering education: review, observations and proposal for global accreditation. European journal of engineering education, 32(6), 639-651. http:// dx.doi.org/10.1080/03043790701520594 Rajaee, N., Junaidi, E., Taib, S. N. L., Salleh, S. F., & Munot, M. A. (2013). Issues and challenges in implementing outcome based education in engineering education. International Journal for Innovation Education and Research, 1(4), 1-9. http://ir.unimas.my/id/eprint/1168 Rashid, M. H. (2012). The Process of Outcome-Based Education-Implementation, Assessment and Evaluations, ASEE international forum, http://www.asee.org/public/conferences/27/pa-pers/8242/view Sekhar, C. R., Farook, O., & Bouktache, E. (2008). Continuous improvement process based on outcome based education. In Proceedings of the 2008IAJC-IJME International Conference. Accessed March (Vol. 5, p. 2013). http://www.ijme. us/cd_08/PDF/31_ENG107.pdf . Spady, W. G. (1994). Outcome-Based Education: Critical Issues and Answers. American Association of School Administrators. http://files.eric.ed.gov/ fulltext/ED380910.pdf Spady, W. G., & Marshall, K. J. (1991). Beyond Traditional Outcome-Based Education. Educational Leadership, 49(2), 67-72. https://eric. ed.gov/?id=EJ432789 Student Outcomes and Program Educational Objectives. (2015). Retrieved july 7, 2015, from http:/ miamioh.edu: http: //miamioh.edu/cec/academ-ics/departments/mme/about/student-outcomes/ index.html
White, H. (2001). Getting started in problem-based learning. The power of problem-based learning, 69-78. https://books.google.com.pk/books?hl=e n&lr=&id=5gJu7IKBC98C&oi=fnd&pg=PA69 &dq=White,+h.+(2001).+Problem+based+learn ing&ots=tD9n_otAbC&sig=FcjguN714fdt6UN vJFjn1R0AvpQ#v=onepage&q&f=false Volkwein, J. F., Lattuca, L. R., Terenzini, P. T., Strauss, L. C., & Sukhbaatar, J. A. V. Z. A. N. (2004). Engineering change: A study of the impact of EC2000. International Journal of Engineering Education, 20(3), 318-328. http://www.ijee.ie/ articles/Vol20-3/IJEE2497.pdf
Annexure 1. Correlation Matrix
Program L earuLns Outcomes
Course Code
IE-101 Problem Solving for SWC SC NC SC SC SWC KC SWC SWC SWC SWC
Industrial Engineers
EE-102 Workshop Practice SC SC SWC SC SC KC SWC SC KC SWC SWC
ME-191 EaEjueerriE Dirav.ir.s ar.J Graphics SC SC SC NC SC NC NC NC NC NC swc
ME-292 Mechanical reclmolDEy SC SC SC SC swc KC NC NC NC swc NC
MA-19] Calculus SC SC SC SWC SC KC KC KC KC SWC KC
MaJ]92 Dif=a:s:t:al Equations SC SC SC SWC SC KC KC KC KC SWC KC
CS-192 Introduction to SC SC SC SWC SC KC KC KC KC SWC SWC
Cmpdiis
HLT-112 Ii]a_-nic Studies Ethics KC KC KC NC KC KC SC KC SWC SC KC
IE-121 Probability uid Statiitici SC SC SC SC SWC KC KC KC KC SWC KC
EE-122 Computer Aided Diiipi SC SC SC KC SC KC KC SWC KC SC SWC Sl Modeling
IE-231 Engineering XlanaEemetit SC SC SC SWC SWC SWC SC swc swc sc nc
ME- 221 F.M:=nr.tEILilE Mscliamci SC SC SC swc sc kc kc NC KC SWC NC
HU-101 AppLiid Physics SC SC SC SC SWC KC KC KC KC SWC KC
ME-293 \litsrialji fagjapamg
MA-193 AppU-sd L^ie^r Aleebra SC sc sc SWC sc NC NC NC NC swc NC
EE-241 Etl'iij-tEI-tlS' EaOUOC-lCS sc sc sc SWC sc sc NC swc NC NC swc
IE-243 Operations Rjeseajdi EC sc sc SWC sc SWC NC NC NC swc swc
EE-242 MiDuflCtiLnil2, PrOCe55 SC EC EC EC sc SWC NC NC NC swc swc
au-i is Pakistan Studies NC NC NC NC NC NC EC NC EC SWC KC
ME-294 I'.'^-I.^iir- of \iaterialj SC sc sc sc SWC SWC NC NC NC SWC NC
EE-301 Industrial ^-1^-tronjij^ sc sc sc SWC sc NC NC NC NC swc swc;
EE-311 Operations of 1. lanufactunna S^'Eiemi sc sc sc SWC sc SWC NC swc NC SWC swc
ME-3I1 Applied MacMne D== & FEM 2 1 EC sc EC sc sc SWC NC SWC NC swc swc
EE-3I2 EE-312 Metrology £ Statistical Qualiiv Control ? 1 EC sc sc swc sc NC NC NC NC sc swc
EE-3I3 IE-313 ppfanintim Tedmiigiea 2 0 SC sc sc swc sc SWC NC NC NC swc swc
ffi-314 EE-314 Work Study A Methods EaE jaeerLiE SC sc sc SWC sc SWC NC SWC SWC swc swc
ffi-321 Industrial Emulation 2 1 SC sc sc SWC sc SWC NC swc NC sc swc
EE-322 EE-322 Human Factors EaEmeeriiE 2 I SC sc sc SWC SWC sc SWT swc SWC sc swc
EE-323 EE-323 Mana§EmEnt of Engineering Projects 3 € SWC sc SWC SWC sc SWC SWC sc SC swc swc
MA-391 MA-391 Xumeriital Analysis 3 D sc sc sc SWC sc NC NC NC NC swc KC
ffi-324 Planning Bdigdnhqe lu l1, ianufacturixiE 2 0 sc sc sc SWC sc SWC NC SWC NC SWC swc
ffi-325 Industrial Autc nation ar.d Rc-L'jt.-i: sc sc sc swc sc sc NC swc NC sc swc
IE4I1 Desieu of Ej^erunents 3 1 sc sc sc sc sc sc NC swc NC sc swc
IE-412 EE412 Industrial Facilities Desien 2 1 sc sc sc swc sc sc NC SWC NC swc swc
ffi-XXX m-xxx Elective: (HEM) SWC SWC SWC swc sc SWC SWC sc SC swc swc;
EE-XXX EE-XXX Elective EI (Metal mnnitis an suttms analysis sc sc sc swc sc sc NC NC NC swc swc
IE-491 EE-491 Project Phase I SC sc sc sc sc sc sc sc SC sc sc
m-xxx Elective I (OrEinizatioijjkl behavior SWC SWC SWC SWC SWC SWC sc sc SWC SWC SWC
m-xxx IE-XXX Elective I 'Automation and Ctrl) sc sc sc swc sc sc NC swc NC sc swc
ffi-XXX ffi-XXX Elective D (can sc sc sc SWC sc sc NC SWC NC sc swc
EE-XXX m-xxxHflcti-«
EE492 EE492 Project Phase 11 sc sc sc sc sc sc sc sc SC sc sc
SC: Strongly correlated, SWC: Somewhat correlated, NC: Not correlated
