CC BY-NC-ND
47
Multi-Framework Implementation of the Problem Management Process
Ruben Pereira
Assistant Professor, ruben.filipe.pereira@iscte-iul.pt Instituto Universitário de Lisboa (ISCTE-IUL), Avenida das Forças Armadas, 1649-026 Lisbon, Portugal
Isaías Scalabrin Bianchi
External Researcher, isaias.bianchi@gmail.com Institute of Scientific Research, Universidad de LimaAv. Javier Prado Este 4600, Santiago de Surco 15023, Lima, Peru
Virgínia Maria da Silva Araújo
Assistant Professor, varaujo@uatlantica.pt
Atlantica University, School of Management Sciences, Health, IT & Engineering, Fábrica da Pólvora de Barcarena, 2730-036 Barcarena, Portugal
Abstract
Many different information technology frameworks have been proposed to assist organizations implementing information technology. However, these frameworks are complex, difficult to implement and overlap each other turning their simultaneous implementation even more difficult to accomplish by organizations. This study proposes to develop an overlapless maturity model that not only help
organizations dealing with the problems aforementioned. The model was applied and evaluated by experts in five organizations. The artefact was recognized as useful, complete and helpful in a multi-framework implementation by PM experts. This research provides contributions for academics since it distinguishes itself from the existing artefacts in the body of knowledge and is a baseline for further investigation.
Keywords: IT frameworks; problem management; Maturity model; multi-framework implementation; frameworks overlap
Citation: Pereira R., Bianchi I.S., da Silva Araújo V.M. (2021) Multi-Framework Implementation of the Problem Management Process. Foresight and STI Governance, 15(3), 94-104. DOI: 10.17323/2500-2597.2021.3.94.104
© 2021 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Introduction
Since the 1980s organizations have sought out competitive advantage and have become more cost-effective through the achievement of operational improvements [Kappelman et al, 2019]. The presence of computer and information technology (IT) in today's organizations has expanded dramatically [Carvalho et al., 2019a; Patón-Romero et al., 2018] and has pushed IT functions to become more service-oriented to be more cost-effective and aligned with business objectives [Carvalho et al., 2019b; Tan et al., 2010]. Nowadays, most organizations are deeply dependent upon IT in order to plan, design, deliver, operate, and control IT services offered to customers. IT departments are actually one of the most complex parts of an organization [Ayat et al., 2009]. To deal with the increase of IT complexity, many IT frameworks have been developed and proposed. All these frameworks have value to offer along with different strengths and weaknesses [Aguiar et al., 2018]. For example, the IT Infrastructure Library (ITIL) [Long, 2008], Control Objectives for Information and Related Technologies (COBIT) [ISACA, 2012], and Capability Maturity Model Integration for Services (CMMI-SVC) [SEI, 2010] are among the most popular ones.
Over the years, organizations have focused heavily on improving their IT processes to be able to bring remarkable benefits. One of the ways to improve IT processes is using IT frameworks and many organizations use them. Some researchers have reported the benefits of these frameworks [Huygh et al., 2018]. However, not only are IT frameworks seen as complex [Serenko et al., 2016], but the lack of assistance for customizing and implementing such frameworks make it difficult for organizations to choose one since it is unclear which IT framework better suits established IT environments [de Haes, van Grembergen, 2017]. Often the processes end by not being consistent and properly defined [Rohloff, 2008]. Plus, most of these IT frameworks overlap each other [de Haes et al., 2013]. This implies a duplication of investment, costs, and human resources for organizations [Gama et al., 2013]. However, they can be combined to narrow the gaps and then become more powerful than individual systems [Aguiar et al., 2018].
As pointed out by several authors such as [Aguiar et al., 2018; Schlarman, 2007] IT frameworks can easily overlap one another. Moreover, IT frameworks are complex to understand and implement [de Haes et al., 2013; Evelina et al., 2010; Herrera, Hillegersberg, 2019; Serenko et al., 2016].
By way of response, the maturity model (MM) concept was introduced to assess the level of a process [Becker et al., 2009]. Process MMs are being implemented by an increasing number of organizations [ Uskarc, Demirors, 2017] because they lay the groundwork as a measure to evaluate an organization's capabilities in a specific discipline [de Bruin et al., 2005]. As pointed out by [Beck-
er et al., 2009], most MMs are considered too general and are usually not clearly defined and documented. Moreover, the current MMs do not address the overlap issue identified by several research studies [Sahibudin et al., 2008]. Among the most important processes proposed by IT frameworks is Problem Management (PM). However, implementing the PM process properly can be complex, long, expensive, and may cause PM implementation to fail [Sharifi et al., 2009; Pereira et al., 2012] leading to low quality service delivery. This means that PM can shape how customers see the entire organization and impact business considerably. Despite the popularity of some IT frameworks, there has been little academic research published to date about issues related to maturity model adoption and implementation [Cater-Steel et al., 2006; Jansen, 2020]. In accordance with the statements mentioned above, this study aims to develop an overlap-less and complete IT MM for the PM process grounded in ITIL, COBIT, and CMMI-SVC content following the theory proposed by [Becker et al., 2009] (which is based on the Design Science Research (DSR) methodology [Gregor, Hevner, 2013]).
Theoretical Background
IT Frameworks
The literature describes many IT frameworks, which are also called best practices and standard guidelines that assist the organization in the management of its technology infrastructure. The most relevant guidelines with the majority of citations are ITIL, COBIT and CMMI-SVC [Pereira, Mira da Silva, 2012]. ITIL is a set of best practices [Long, 2008] and one of the world's most widely accepted approaches to ITSM [Saarelainen, Jantti, 2016]. ITIL necessitates cultural changes and usually requires the use of specialists to enact and adapt it to each organizational context [Bo-vim et al., 2014]. The ITIL framework has been adopted by companies of all sizes and industries, including large, medium, and small businesses. COBIT is an IT framework for designing, adopting, tracking, and optimizing IT governance and management procedures. It is one of the most widely used in the world. Its processes are divided into governance and management areas.
Finally, the CMMI-SVC [SEI, 2010] not only defines IT procedures but also a set of practices and goals that companies can use to implement their own sets of processes. A particular objective, according to the CMMI framework, defines the unique features that must be present to meet the process requirements. A specific practice is a description of an activity that is thought to be critical in achieving the associated goal [Aguiar et al., 2018]. Plus, this section also intends to present a brief analysis of each IT framework (Table 1). Since the PM process is included in each framework, it makes them suitable IT frameworks upon which to ground our proposal.
IT Maturity Models
There is a consensus in the literature regarding the efficiency of MMs (Table 2). MMs depict a hierarchy of maturity levels for a certain class of objects (typically organizations or processes [Becker et al., 2009]) and the expected, required, or typical evolution paths of these objects in the form of discrete stages. This format allows for evaluating the applied processes through the prism of best management experience and a set of external parameters.
The initial work from AXELOS to measure service management processes started with the Process Maturity Framework (PMF) which was published and made available as an appendix of the ITIL "Service Design" publication [Long, 2008]. This PMF is used as a framework to evaluate the maturity of each of the Service Management processes independently or to measure the maturity of the overall Service Management process. [Long, 2008]. An updated model named "ITIL Maturity Model" presumes a self-assessment service to help organizations improve their IT service management within the ITIL framework [Aguiar et al., 2018]. This self-assessment is based on a series of questionnaires for each process and function in the ITIL service lifecycle.1
Following the research conducted by [Aguiar et al., 2018] as a reference, Tables 2 and 3 provide a short summary of the previously mentioned IT MMs so that readers can better understand how these MMs differ from one another. Almost all of the compared MMs have five levels. Two of them base their theories on ISO/IEC 15504. The most intriguing fact is that each of the identified MMs takes a unique approach, focusing solely on their own theory. It should be observed that none of these MMs address the problem of IT framework overlap.
One of the main advantages of the proposed approach is that the person doing the evaluation does not need to be an IT governance specialist because the analysis is done automatically [Simonsson et al., 2007]. The modeling language is based on COBIT and provides for the identification of entities and relations. The entities identified were: activities, processes KPI/KGI, documents, and roles [Aguiar et al., 2018]. Pereira and Mira da Silva [Pereira, Mira da Silva, 2010, 2011] proposed a model that is also based on CMMI-SVC. This MM was distinguished among others on the market at the time because it was designed exclusively to assist businesses in measuring their ITIL v3 maturity and leading them through the implementation of ITIL. The proposed IT Service Delivery MM, on the other hand, was a mechanism for formalizing and assessing IT Service Delivery Elements [Flores et al., 2011]. The authors of the aforementioned study established five levels of maturity, similar to CMMI-SVC. The adopted scale to score the maturity level is 1 to 5. To better distinguish between maturity states, the authors add
a "+" or a "-" whether the level is closer to being up or down. Vitoriano and Neto [Vitoriano, Neto, 2016] used a methodology based on the Process Maturity Framework (PMF), an MM defined in the ITIL (v2) reference model. To use this MM, some interviews with questions related to the five maturity levels, such as initial, repetitive, defined, managed, and optimized, are required; information was gathered on five fundamental ITSM processes.
More recently Aguiar et al. [Aguiar et al., 2018] developed a MM for the incident management (IM) process where the overlap issue was addressed and mitigated. The authors also took into consideration the main IT frameworks of the market. The results were exciting with great feedback from the experts. The study found that the main IT frameworks overlap each other almost 25% regarding the IM process. After analyzing the main IT frameworks and MMs among the literature, the authors were able reinforce the theory that most MMs ground their development on CMMI. Moreover, only one of the analyzed MMs take into consideration the overlap issue. It is the most recent study [Aguiar et al., 2018] where the researchers developed a MM for the IM process and incentivized future researchers to develop overlap-less MMs for the remaining IT processes. Therefore, such findings strengthen the aim and relevance of this research. It can be observed that the inquiry into the implementation of multi-frameworks and how it can be handled and measured has been financially rewarded [de Haes et al., 2013].
Research Methodology
Recently, Design Science Research (DSR) has gained importance and popularity in information systems. Many researchers have used DSR to develop an innovative artefact in order to solve a specific and relevant organization problem domain [Hevner et al., 2004]. The adopted research methodology was the DSR which has been incentivized to be used in a myriad of fields [Rai, 2017] including IT Governance, covering a broad range of IT-related processes [Gregor, Hevner, 2013; de Maere, de Haes, 2017]. The key elements of the DSR under investigation are the possibilities of discovering new fields of research, conducting testing and the validation of theories, or building new theories. The purpose of this work is to to develop an overlap-less maturity model to solve a specific problem and help the organizatons. Therefore, DSR can be a suitable approach for this study. The proposed artefact was designed and evaluated following Peffers guidelines [Pef-fers et al., 2007] as you can see in the Figure 1.
Proposal of an Overlap-less Maturity Model
For the development of MMs, Becker et al. [Becker et al., 2009] identified a set of necessary requirements with which our proposal strictly complies (Table 4). In
1 https://docplayer.net/655929-Itil-maturity-model-october-2013.html, accessed 15.07.2021.
Table 1. Comparing IT Frameworks
Model ITIL V3 COBIT 5 CMMI-SVC
Founder OGC ISACA, ITGI Software Engineering Institute (SEI)
Focus IT Service IT Service IT Service
PM Yes Yes Yes
Name of Process Problem Management Manage Problems Causal Analysis and Resolution
Number of Processes 26 37 24
Source: authors.
addition, the development of the proposed PM MM was accomplished by following three steps: (1) Elici-tation of PM activities from the most well-known IT frameworks; (2) Elimination of overlaps; and (3) definition of the maturity level for each elicited activity. Phase 1: The first step focused on identifying all of the PM activities present in the ITIL, CMMI-SVC, and COBIT frameworks, as well as specifically identifying the IT frameworks supporting each elicited activity (Table 5). At the end of this phase, 349 activities had been gathered (Table 6). Table 7 shows a sample of its activities. The authors went through four iterations of fine-tuning the list to reach at a consensus final list. Phase 2: The authors using the initial list concluded the phase (ant-overlap), moved on to the next, which involved a thorough identification of IT framework overlaps. During this phase, all activities were separated by process areas to make identifying overlaps easier. To demonstrate the outcome of this step, the authors present Table 7, which explains how the overlap elimination was carried out. By the end of this phase, 46 PM activities had been identified as overlapping among the selected IT frameworks. This accounts for 13% of the initial set of activities gathered. It was possible to create a new list (post-overlap) of activities with 303 activities by merging activities and eliminating overlaps (Table 7).
Finally, to complete the proposal, the authors organized the final set of activities by maturity level. The maturity levels were assigned based on the adherence
Table 2. Comparison of Frameworks' MMs
Model COBIT PAM CMMI-SVC AXELOS
Number of levels 0-5 SM:1-5 1-5
Scope Governance CM:0-5 IT Services
Based on ISO/IEC 15504 IT Services —
Approach Individual — Individual
Frameworks overlap Not addressed Individual Not addressed
I Source: authors.
of each activity to the CMMI-SVC description of maturity levels.
Using the same activities as in Table 5, the authors present Table 7 to illustrate how the maturity levels were assigned to each activity.
As an example, only one activity sample was provided for each existing maturity level. Here, the authors decided to follow the maturity level definitions of CMMI-SVC since they are used in the development of most MMs present in the literature. An activity classified as level 2 is considered a basic activity in the PM process since it is the first step for information collection. An activity classified as level 3 is mostly included among standards, procedures, or methods. An activity ranked as level 4 is focused on process measurement; such are usually metrics aimed at measuring a specific process aspect. Finally, an activity classified as level 5 is focused on the continuous improvement of processes and all activities involved in pursuing this kind of activity type. During the semi-structured interviews, a questionnaire was provided that consisted of all collected post-overlap activities, arranged by order of process (problem identification, problem logging, etc.), in order to become rational and concise throughout its course.
It should be stated that this artefact only focuses on the framework activities. There are other relevant concepts (for example inputs, outputs, metrics, etc.) that organizations must still have to collect from the IT frameworks. However, by using our artefact first, organizations will have a clear vision of the core activities and the respective frameworks that they can then check later for further information. Our artefact does not substitute the IT frameworks. It may be seen as a complement to guiding organizations in further steps.
Demonstration and Evaluation
In order to demonstrate the proposed artefact, the authors have searched for organizations with PM processes in place (up and running) and invited them to participate. Five organizations accepted the invitation to be assessed by the authors and to evaluate the proposed artefact. Both demonstration and evaluation were performed through semi-structured interviews
Table 3. Comparison of MM Levels
Level COBIT PAM CMMI-SVC AXELOS
0 Incomplete — —
1 Performed Initial Initial
2 Managed Managed Repeatable
3 Established Defined Defined
4 Predictable Quantitatively Managed Managed
5 Optimizing Optimizing Optimizing
6 — — —
Source: authors.
Figure 1. DSR Phases
Problem identification and motivation
IT frameworks overlap each other
IT frameworks are complex to understand and implement
>
Definition of the objectives for a solution
The artefact must be able to eliminate framework overlap through multi-framework implementation
>
Design and Development
Identify PM activities from
the most well-known IT
frameworks
Eliminate overlaps
Assign a maturity level to each
collected activity
Design a MM for PM
>1
Demonstration
Use the MM to assess PM in different organizations
Evaluation
Semi-structured
Communication
Submission to reference and respectful journals
Source: adapted from [Peffers et al., 2007].
with experts from the respective organizations. In particular, the authors interviewed the PM process owner of each organization. During each interview, the list (post-overlap) was presented to the interviewee so he could confirm whether each activity had been implemented or not. The maturity level of each activity was not presented to avoid biased answers. At the end, the individual and global reports were sent to the interviewee. Any person/organization intending to apply the artefact in the future should perform it in the same way.
Data Collection and Analysis
The interviews were conducted in different organizational contexts and with the most suitable decision-makers to assess and provide information about the PM process. Details about the interviewees can be found in Table 8.
The interviews were conducted between March and July of 2018. In a total of five interviews (two via Skype and three in person). The average time required for
each interview was one hour and 45 minutes. To prepare and help the interviewees before the assessment, a questionnaire was developed and delivered a few days before the interview. The questionnaire to frame the interview was developed in three parts. The first part contained general questions about the organization and the profile of interviewee. The second delved into the implementation of the activities. Finally the third part posed questions about the points of view and opinions of the interviewee regarding the PM MM. In Table 9 one can see organization's details. Organizational culture was described based on the theory proposed by [Matthyssens, Wursten, 2002]. Overall the assessed organizations have at least 1,400 employees and considerable IT Departments. Some organizations did not allow the publication of some information. None of the assessed organizations had any sense of their maturity level. Such evidence brings even more relevance to this study. According to [Pereira, Mira da Silva, 2012], in order to achieve a maturity level, organizations had to imple-
Table 4. How the Proposed Artefact Complies with the Becker Requirements
Requirement Description
Comparison with existing MMs A comparison between IT frameworks should be made, mainly focusing on the most well-known and relevant for the case.
Iterative procedure The identification of the first list of activities (1) was achieved through an iterative process. Plus, interviews can be considered an interaction due the continuous feedback received from practitioners in order to improve the artefact.
Evaluation For the assessment of the artefact, five semi-structured interviews were performed keeping in mind the interactive process used in all interviews.
Multi-metho dological procedure Several methodologies were used for model creation: literature review, cross frameworks analysis, and semi-structure interviews. Plus, this research fulfils DSR procedures and Becker requirements.
Problem definition There is no limitation in the application of the proposed PM MM unless PM practices already exist at the target organization. It can be applied at any organization regardless of its classification presented in [Pereira et al., 2013]. The main expected benefit is the prior identification of overlapping activities that may save resources in future implementations of multi-frameworks.
Interim monitoring and target presentation of results Based on results collected throughout the assessment of the artefact, it is possible to provide two types of reports: an individual report for each organization and a global/cross-organization report. The individual report can provide information regarding current organization maturity level, a maturity roadmap including the required steps to reach the next level. Information can also be found about achieved activities and the identification of which framework complies best as well as missing activities identified in the roadmap. By using the roadmap, organizations are able to become more efficient at saving resources in future multi-framework implementations. The global report is achievable by combining and cross-referencing all information received from each assessment.
Source: authors.
Table 5. Sample of Pre-Overlap Activities among IT Frameworks
Activity IT Framework
Is the defect or problem identified? COBIT
Is a problem record raised? If yes, does the problem contains all relevant details? ITIL
After the problem is identified, do you usually develop a suitable workaround? CMMI-SVC
Do you usually analyze the change in process performance of the affected processes or sub-processes for the work? If yes, do you measure it? COBIT
Are the lessons learned from the review presented at a service review meeting with the business customer? ITIL
Do you usually try to find a workaround to temporarily solve the problem? ITIL
Is the problem identified? CMMI-SVC
Source: authors.
ment at least 75% of the activities of that corresponding level. Based on Figure 2, one can see level 2 is the most mature among the assessed organizations, followed by level 3, level 4, and level 5, respectively. Overall organizations are more focused on definition and management activities but neglect metrics and measures to promote continuous improvement and predictive analysis.
An individual analysis is presented in Figure 3. All organizations have a similar maturity level, generally at level 2 (Managed). Level 5 (optimizing) is the lowest level, followed by level 4 (Quantitatively Managed) and finally level 3 (defined). Apparently, there is no visible disparity between the various types of organizations. Despite the authors' conviction, none of the assessed organizations met the conditions to reach level 2 (75%). The telecommunications company is the nearest one to achieve it. All organizations are at level 1 (initial). On average, the organizations tended to focus their efforts toward the first two levels, level 2 and 3. To be considered a managed process (level 2) and reach level 3 (defined), most organizations would need to implement
Table 6. PM Activity Results after Applying the First Two Steps
Model PM process name Number of activities (n°) Percentage (%)
ITIL Problem Management 153 44
COBIT Manage Problems 85 24
CMMI-SVC Causal Analysis and Resolution 111 32
Ant-Overlap activities 349 100
Overlapped activities 46 13
Post-Overlap activities 303 87
Source: authors.
between 12% and 37% of the remaining activities. For some, it may be a considerable effort. Overall, the software organization seems to be the least mature and the bank seems to be the most mature. The assessed bank is the only one with a similar percentage for levels 2 and 3. All the other organizations have a considerable higher percentage of level 2. The telecommunications company achieved the highest percentage for level 2 but falls about 20% when considering level 3 while the bank has a more stable and balanced percentage among the first two levels.
Another interesting finding is that, apparently, organizations are aligned with MM theory. According to the MM theory, a previous level is crucial for achieving the next level. This means that it would not make sense, for example, to have a higher percentage of level 3 than level 2. Based on that, the authors may argue that organizations are aligned with these guidelines. None of the organizations have a maturity level with higher percentage than the previous one. Such a fact indicates that despite none of the assessed organizations being at level 2, they are implementing the process in a coordinated and balanced way.
Additional insights can be obtained regarding the IT frameworks adopted within each organization. Most of the interviewed organizations (80%) pointed to ITIL as the officially adopted IT framework with the last organization adopting CMMI-SVC (20%). Such a finding is aligned with previous studies claiming that ITIL was one of the most adopted IT frameworks on the market [Long, 2008; Saarelainen, Jantti, 2016]. Plus, the authors also found that ITIL activities are the most implemented in number and percentage. Table 10 illustrates all the insights gathered from the assessments regarding the adoption of each IT framework.
Evaluation
After completing the interview process, the interviewees were invited to provide some feedback by answering some questions in order to evaluate the appraoch and consequently the problem statement of our research. As illustrated in Table 11, from a global perspective, the opinion was positive. Some interviewees mentioned that it was exhaustive but complete which is quite understandable. However, it was agreed upon among the interviewees that the proposed artefact is useful providing a complete vision of the PM process based on the three most-known IT frameworks. No activity was considered absent, which validates the first (1) and second (2) steps performed to develop the artefact.
Discussion
Despite the existence of several IT frameworks to help organizations to increase IT efficiency, such are seen as complex [Serenko et al., 2016], difficult to implement [de Haes, van Grembergen, 2017], seen to overlap one another [Schlarman, 2007; Pereira, Mira da
Table 7. Demonstration of the Merging Process
Activity Maturity Level ITIL CMMI-SVC COBIT
Are the problems identified? 2 Is a problem record raised? Is the defect or problem identified? Is the problem identified?
Do you usually try to find a workaround to solve the problem? 3 Do you usually try to find a workaround to temporarily solve the problem? After the problem is identified, do you usually develop a suitable workaround?
Do you usually analyze the change in the performance of the affected processes or sub-processes for the work? If yes, do you measure it? 4 Do you usually analyze the change in the performance of the affected processes or sub-processes for the work? If yes, do you measure it? —
Are the lessons learned from the review presented during a service review meeting with the business customer? 5 Are the lessons learned from the review is presented during a service review meeting with the business customer? —
Source: authors.
Table 8. Details about the Interviewees
Country Position Experience in IT (years) Duration of interview (H) Procedure
Portugal IT Manager 18 2h40 Face to face
Portugal IT Team Leader 8 1h50 Face to face
USA Application Support Lead 12 1h30 Virtual
Portugal IT Director 16 1h12 Virtual
Portugal IT Director 20 1h20 Face to face
Source: authors.
Table 9. Factor Analysis and Details about the Interviewee's Organization
Industry Size IT Employees Market IT Strategy IT Structure Culture
Telecommunication 2100 400 Worldwide Flexibility Decentralized Pyramidal
Energy, Automation and Telecommunication 1400 28 Worldwide Flexibility Decentralized Pyramidal
Pharmaceutical 42 000 1300 Worldwide Efficiency Federal Contest
Software 13 000 — Worldwide — — —
Banking — — Wordwide Flexibility Federal Pyramidal
Source: authors.
Figure 2. Average Implemented Activities (%)
а) percentage
b) number
70 60 50 40 30 20 10 0
63
Level 2 Level 3 Level 4 Level 5
140 120 100 80 60 40 20 0
Achieved
■ Total
49
18
■ 4 1
Level 4 Level 5
Source: authors.
Figure 3. Maturity Level of Each Organization (activities achieved by type of organization)
Telecommunication
Energy, Automation and Telecommunication
Pharmaceutical Software
Source: authors.
Silva, 2011], and generic [Pereira, Mira da Silva, 2012]. Therefore, this research proposes an artefact that mitigates some of the existing gaps in multi-framework implementation such as framework overlap and complexity. This research not only confirms the gaps found earlier but aim to solve them with the proposed artefact which may help organizations in multi-framework implementations. All the interviewees found the artefact useful (demonstrated in practice) and complete (no activity was thought to be missing). According to [de Haes et al., 2013; Aguiar et al., 2018], multi-framework implementation is a real challenge. Many organizations are not yet aware of their implementation and fail to yield the best results from them. A PM MM was then developed by the authors merging all details and knowledge of the three most well-known IT frameworks on the market regarding PM process. During the initial process in the creation of the model, the three most well-known and used IT frameworks (COBIT, CMMI-SVC and ITIL) were analyzed. In the end this research confirmed the existence of overlaps
Table 11. PM MM Evaluation
Interview number Completeness Missing activities Usefulness
1 Too long / Overtired No Yes
2 Very complete No Yes
3 Yes No Very
4 Very No Yes
5 Yes No Yes
Source: authors.
between IT frameworks. About 13% of the elicited PM activities were common to at least two IT frameworks. This research provides novel insights for academics given that a new artefact absent from the literature was developed merging all the main IT service management frameworks regarding the Problem Management process and tested at real organizations. This may now be assumed as a base for further investigation for the remaining IT service management processes. This research also contributes to the performance of professionals since they now have a tool to assess their Problem Management process maturity. It will help them achieve higher levels of maturity and be aware of current overlaps. Consequently, they may save resources that can be allocated to other processes.
Adding Knowledge by Crossing Studies
Cross-referencing similar studies and findings is an interesting exercise that can be used to evolve the body of knowledge and bring new insights to the scientific community. As previously stated, a similar study [Aguiar et al., 2018] was performed in the past but focused on the IM process instead of PM. The current research was also motivated by the future work proposed by the previous investigation. Table 12 and Table 13 present the information combined from both studies. It is interesting to note that in both studies (Table 12) the highest overlap percentage belongs to the activities common to the three IT frameworks while the lowest belongs to the activities common between COBIT and CMMI-SVC. On the other hand, other findings can be drawn from Table 13. It seems that, when looking for
Table 10. Analysis of the Adoption of Each IT Framework within the Model
Models ITIL CMMI-SVC COBIT ITIL& CMMI-SVC ITIL& COBIT CMMI-SVC & COBIT All Total
Overall activities (number) 101 89 73 7 11 3 19 303
Overall activities (%) 33.33 29.37 24.09 2.31 3.63 0.99 6.27 100
PM process overlap (%) 2.31 3.63 0.99 6.27 13.20
Average implemented activities (number) 90 72 65 4 9 2 15 257
Average/Total implemented activities (%) 29.70 23.76 21.45 1.32 2.97 0.66 4.95 —
Average/Overall implemented (%) 89.11 80.90 89.04 57.14 81.82 66.67 78.95 —
Source: authors.
Table 12. Cross-Study: Overlapped Activities (%)
Overlapped Activities ITIL & CMMI SVC ITIL & COBIT CMMI-SVC & COBIT All Total
Incident Management 5.3 2.4 1.4 14.5 23.6
Problem Management 2.31 3.63 0.99 6.27 13.20
Note: the colored cells in dark and light represent the minimum and maximum of some specific frameworks. Source: authors.
Table 13. Cross-Study: Implemented Activities (%)
Implemented Activities ITIL CMMI-SVC COBIT ITIL& CMMI-SVC ITIL& COBIT CMMI-SVC & COBIT All
Incident Management 70.1 79.0 72.2 77.4 62.5 71.4 84.1
Problem Management 89.11 80.90 89.04 57.14 81.82 66.67 78.95
both processes, organizations have different preferences regarding which IT frameworks to implement.
Conclusion
This research aims to create an artefact to help organizations in multi-framework implementation by eliminating overlaps among IT frameworks. To do it, the authors chose one of the most relevant IT processes (PM) and developed an overlap-less PM MM. The validity of the artefact was confirmed by applying and evaluating it at five different organizations. This research confirms and reinforces the issue of IT framework overlaps previously identified by other researchers. From the 349 PM activities elicited, 46 activities were identified as being areas of overlap among the chosen IT frameworks. Almost 15% of all activities are present in at least two of the three IT frameworks analyzed in this research (Table 10). All the interviewees considered the artefact useful and complete. They confirmed that implementing an IT framework is not straightforward and having a method to help them in multi-framework implementation would be very useful [de Haes et al., 2013; Aguiar et al.,
2018]. By recognizing the proposed artefact as complete, the interviewees (PM experts) accept it as helpful. Looking at the assessed organizations, four of them (80%) pointed to ITIL as the official IT framework. The fifth organization (20%) adopted CMMI. It is interesting to note that despite none of the assessed organizations reaching level 2, they have been implementing the PM process in a balanced way. From a cross studies analysis, both processes (IM and PM) tend to have the highest percentage of activities overlapped by all the IT frameworks and the lowest percentage of activities common between COBIT and CMMI-SVC.
This research also has some limitations. The authors think that the previous conclusions may change when considering the remaining IT processes and within different organizations contexts. Built on such limitations, future work may assess the model at more organizations and consequently develop similar MMs for the rest of the existing IT processes. It is also authors' conviction that, having an integrated model able to cover most of the adopted IT processes could be very useful and at the same time challenging, which stands as a reason to continue this research.
References
Aguiar J., Pereira R., Vasconcelos J. B., Bianchi I. (2018) An Overlapless Incident Management Maturity Model for MultiFramework Assessment (ITIL, COBIT, CMMI-SVC). Interdisciplinary Journal of Information, Knowledge, and Management, 13, 137-163. https://doi.org/10.28945/4083
Ayat M., Sharifi M., Sahibudin S., Ibrahim S. (2009) Adoption factors and implementation steps of ITSM in the target organizations. Paper presented at the 2009 Third Asia International Conference on Modelling & Simulation 25-29 May 2009 Bundang, Indonesia. https://doi.org/10.1109/AMS.2009.114
Becker J., Knackstedt R., Pöppelbuß J. (2009) Developing Maturity Models for IT Management. Business & Information Systems Engineering, 1(3), 213-222. https://doi.org/10.1007/s12599-009-0044-5
Bianchi I.S., Sousa R.D., Pereira R. (2017) IT Governance Mechanisms at Universities: An Exploratory Study. Paper presented at the Twenty-third Americas Conference on Information Systems, August 2017, Boston, USA. https://www.researchgate.net/ publication/319141074_IT_governance_Mechanisms_at_Universities_An_Exploratory_Study, accessed 16.05.2021.
Bovim A., Johnston K., Kabanda S., Tanner M., Stander A. (2014) ITIL adoption in South African: A Capability Maturity view. Paper presented at the E-Skills for Knowledge Production and Innovation Conference 2014, Cape Town, South Africa. http://proceedings.e-skillsconference.org/2014/e-skills049-060Bovim759.pdf, accessed 17.06.2021.
Carvalho J.V., Rocha Á., Abreu A. (2019a) Maturity Assessment Methodology for HISM M — Hospital Information System
Maturity Model. Journal of Medical Systems, 43(2), 35. https://doi.org/10.1007/s10916-018-1143-y Carvalho J.V., Rocha Á., van de Wetering R., Abreu A. (2019b) A maturity model for hospital information systems. Journal of
Business Research, 94, 388-399. https://doi.org/10.1016/JJBUSRES.2017.12.012 Cater-Steel A., Tan W.-G., Toleman M. (2006) Challenge of adopting multiple process improvement frameworks. Paper presented at the 14th European Conference on Information Systems, ECIS 2006, Göteborg, Sweden, 2006. https://www.researchgate. net/publication/221408576_Challenge_of_adopting_multiple_process_improvement_frameworks, accessed 10.05.2021.
De Bruin T., Freeze R., Kulkarni U., Rosemann M., Freeze R., Carey W.P. (2005) Understanding the Main Phases of Developing a Maturity Assessment Model. Paper presented at the 16th Australian Conference on Information Systems (ACIS 2005). http://aisel.aisnet.org/acis2005, accessed 08.06.2021.
De Haes S., van Grembergen W. (eds.) (2017) Strategic IT governance and alignment in business settings. Hershey, PA: IGI Global. https://doi.org/10.4018/978-1-5225-0861-8
De Haes S., van Grembergen W., Debreceny R.S. (2013) COBIT 5 and Enterprise Governance of Information Technology: Building Blocks and Research Opportunities. Journal of Information Systems, 27(1), 307-324. https://doi.org/10.2308/isys-50422
De Maere K., De Haes S. (2017) Is the Design Science Approach fit for IT Governance Research? Paper presented at the 16th European Conference on Research Methodology for Business and Management Studies, June 2017, Dublin. https://www. proquest.com/openview/d898b4819205b18146e1bddd65bd615a/1?pq-origsite=gscholar&cbl=1796413#:~:text=The%20 results%20suggest%20that%20top,elements%20in%20their%20research%20design, accessed 24.01.2021. Evelina E., Pia G., David H., von Würtemberg L.M., Waldo R.F. (2010) Process improvement framework evaluation. Paper presented at the 17th International Conference on Management Science & Engineering, 24-26 November 2010, Melbourne, VIC, Australia. DOI: 10.1109/ICMSE.2010.5719823. Flores J., Rusu L., Johannesson P. (2011) A Maturity Model of IT Service Delivery. Paper presented at the International Conference on Information Resources Management (CONF-IRM), June 2011. https://aisel.aisnet.org/cgi/viewcontent. cgi?article=1009&context=confirm2011, accessed 12.06.2021.
Gama N., Sousa P., da Silva M.M. (2013) Integrating Enterprise Architecture and IT Service Management. In: Building Sustainable Information Systems (eds. H. Linger, J. Fisher, A. Barnden, C. Barry, M. Lang, C. Schneider), Heidelberg, Dordrecht, London, New York: Springer, 153-165. https://doi.org/10.1007/978-1-4614-7540-8_12
Gregor S., Hevner A.R. (2013) Positioning and Presenting Design Science Research for Maximum Impact. MIS Quarterly, 37(2), 337-355. https://doi.org/10.2753/MIS0742-1222240302
Herrera M., van Hillegersberg J. (2019) Using Metamodeling to Represent Lean Six Sigma for IT Service Improvement. Paper presented at the 21st IEEE Conference on Business Informatics (CBI), 15-17 July 2019, Moscow, Russia. https://doi. org/10.1109/CBI.2019.00034
Hevner A.R., March S.T., Park J., Ram S. (2004) Design Science in Information Systems Research. MIS Quarterly, 28(1), 75-105. https://doi.org/10.2307/25148625
Huygh T., De Haes S., Joshi A., van Grembergen W. (2018) Answering key global IT management concerns through IT governance and management processes: A COBIT 5 view. Paper presented at the 51st Hawaii International Conference on System Sciences. DOI: 10.24251/HICSS.2018.665 ISACA (2012) COBIT 5: Enabling Processes, Schaumburg, IL: Information Systems Audit and Control Association. ISACA (2013) COBIT 5: Process Assessment Model (PAM), Schaumburg, IL: Information Systems Audit and Control Association.
Jansen S. (2020) A focus area maturity model for software ecosystem governance. Information and Software Technology, 118,
106219. https://doi.org/10.1016/jj.infsof.2019.106219 Kappelman L., Johnson V., Torres R., Maurer C., McLean E. (2019) A study of information systems issues, practices, and leadership in Europe. European Journal of Information Systems, 28(1), 26-42. https://doi.org/10.1080/0960085X.2018.1497929 Long J.O. (2008) ITIL® Version 3 at a Glance: Information Quick Reference, Heidelberg, Dordrecht, London, New York: Springer, https://doi.org/10.1007/978-0-387-77393-3
March S.T., Smith G.F. (1995) Design and natural science research on information technology. Descision Support Systems, 15(4), 251-266. https://doi.org/10.1016/0167-9236(94)00041-2
Matthyssens P., Wursten H. (2002) Internal Marketing. Cross-cultural marketing (eds. R. Rugimbana, S. Nwankwo), Toronto, ON: Thompson Educational Publishing, pp. 243-256.
Patón-Romero J.D., Baldassarre M.T., Rodríguez M., Piattini M. (2018) Green IT Governance and Management based on ISO/
IEC 15504. Computer Standards & Interfaces, 60, 26-36. https://doi.org/10.1016ZJ.CSI.2018.04.005 Peffers K.E.N., Tuunanen T., Rothenberger M.A., Chatterjee S. (2007) A Design Science Research Methodology for Information
Systems Research. Journal of Management Information Systems, 24(3), 45-77. https://doi.org/10.2307/40398896 Pereira R., Almeida R., Mira da Silva M. (2013) How to Generalize an Information Technology Case Study. Paper presented at the 8th Design Science Research in Information Systems and Technology (DESRIST), June 11-12, 2013, Helsinki, Finland. https://doi.org/10.1007/978-3-642-38827-9_11 Pereira R., Mira da Silva M. (2010) ITIL maturity model. Paper presented at the 2010 5th Iberian Conference on Information
Systems and Technologies (CISTI), June 16-19, Santiago de Compostela, Spain. Pereira R., Mira da Silva M. (2011) A Maturity Model for Implementing ITILv3. Paper presented at the IEEE 15th International Enterprise Distributed Object Computing Conference Workshops, 29 August - 2 September 2011, Helsinki, Finland. DOI: 10.1109/EDOCW.2011.30
Pereira R., Mira da Silva M. (2012) Designing a new integrated IT governance and IT management framework based on both scientific and practitioner viewpoint. International Journal of Enterprise Information Systems (IJEIS), 8(4), 1-43. https://doi. org/doi:10.4018/jeis.2012100101
Rai A. (2017) Diversity of Design Science Research. MIS Quarterly, 41(1), 3-18. https://aisel.aisnet.org/misq/vol41/iss1/2, accessed 19.11.2020.
Rohloff M. (2008) A Reference Process Model for IT Service Management. Paper presented at the AMCIS 2008 Conference, 14-17 August 2008, Toronto, ON, Canada. http://aisel.aisnet.org/amcis2008/2, accessed 19.11.2020.
Saarelainen K., Jantti M. (2016) Quality and human errors in IT service infrastructures — Human error based root causes of incidents and their categorization. Paper presented at the 11th International Conference on Innovations in Information Technology (IIT), 1-3 November 2015, Dubai, United Arab Emirates. https://doi.org/10.1109/INN0VATI0NS.2015.7381541
Sahibudin S., Sharifi M., Ayat M. (2008) Combining ITIL, COBIT and ISO/IEC 27002 in order to design a comprehensive IT framework in organizations. Paper presented at the 2nd Asia International Conference on Modelling & Simulation (AMS), 13-15 May 2008, Kuala Lumpur, Malaysia. https://doi.org/10.1109/AMS.2008.145
Schlarman S. (2007) Selecting an IT control framework. Information Systems Security, 16(3), 147-151. https://doi. org/10.1080/10658980701225440
SEI (2010) CMMI® for Services, Version 1.3 CMMI-SVC, V1.3 Improving processes for providing better services, Pittsburgh, PA: Software Engineering Institute.
Serenko A., Bontis N., Hull E. (2016) An application of the knowledge management maturity model: The case of credit unions. Knowledge Management Research & Practice, 14(3), 338-352. https://doi.org/10.1057/kmrp.2014.37
Sharifi M., Ayat M., Ibrahim S., Sahibuddin S. (2009) The most applicable KPIs of problem management process in organizations. International Journal of Simulation: Systems, Science and Technology, 10(3), 77-83. http://ijssst.info/Vol-10/ No-3/paper8.pdf, accessed 22.06.2021.
Simonsson M., Johnson P., Wijkström H. (2007) Model-Based IT Governance Maturity Assessments with Cobit. Paper presented at the 15th European Conference on Information Systems, ECIS 2007, St. Gallen, Switzerland. http://aisel.aisnet. org/ecis2007, accessed 17.11.2020.
Tan W.-G., Cater-Steel A., Toleman M. (2010) Implementing IT service management: A case study focusing on critical success factors. Journal of Computer Information Systems, 50(2), 1-12. https://doi.org/10.1080/08874417.2009.11645379
Uskarci A., Demirörs O. (2017) Do staged maturity models result in organization-wide continuous process improvement? Insight from employees. Computer Standards and Interfaces, 52, 25-40. https://doi.org/10.1016/jxsi.2017.01.008
Venable J.R., Pries-Heje J., Baskerville R. (2017) Choosing a Design Science Research Methodology. Paper presented at the 28th Australasian Conference on Information Systems, 4-6 December 2017, Hobart, Australia. http://hdl.handle. net/20.500.11937/70119, accessed 18.05.2020.
Vitoriano M.A.V., Neto J.S. (2016) Information technology service management processes maturity in the Brazilian Federal direct administration. Journal of Information Systems and Technology Management, 12(3), 663-686. https://doi.org/10.4301/ S1807-17752015000300009
Webster J., Watson R.T. (2002) Analyzing the Past to Prepare for the Future: Writing a Literature Review. MIS Quarterly, 26(2), 13-23. https://doi.org/10.L1.104.6570
