CC BY-NC-ND
31
Roadmapping in the Era of Uncertainty: How to Integrate Data-Driven Methods with Expert Insights
Sungjoo Lee
Professor, Department of Industrial Engineering, [email protected] Ajou University, Worldcup-ro 206, Yeongtong-gu, Suwon-si, Gyeonggi-do, South Korea
Kook Jin Jang
Expert, [email protected]
Myung Han Lee
Expert, [email protected]
Seong Ryong Shin
Expert, [email protected] Hyundai Motor Company, 2 Heolleung-ro, Yangjae-dong, Seocho-gu, Seoul, South Korea
Abstract
Roadmapping has long been regarded as a practical tool for supporting decision-making for science and technology innovation and it has received recent attention for its potential use in responses to uncertainty. Indeed, roadmapping enables forward-looking strategy making and thus helps to reduce uncertainty. Accordingly, numerous studies have been conducted to propose new approaches to roadmapping for a wide range of contexts, including the data-driven and expert-based approaches. Although these two main approaches have distinct advantages and disadvantages, few previous studies have focused on how to integrate them into roadmapping to better support decision-making related to science and technology innovation. To address this research gap, this
study investigated how to integrate data-driven approaches with expert insights during roadmapping. For this purpose, a workshop-based roadmapping method was combined with data-driven methods to test this approach in the context of technology planning for the automobile industry. An ethnographic approach was used to collect data on when, where, and how data analysis must be conducted to support experts' discussions. The research findings open a discussion regarding how to integrate data-driven methods with expert insights during roadmapping based on the trade-offs between the two types of data, that is, hard data for data-driven methods and soft data from expert insights and suggest possible opportunities for future roadmapping developments.
Keywords: roadmapping; uncertainty; data-driven approach; expert insights; foresight
Citation: Lee S., Jang K.J., Lee M.H., Shin S.R. (2021) Roadmapping in the Era of Uncertainty: How to Integrate Data-Driven Methods with Expert Insights. Foresight and STI Governance, 15(2), 39-51. DOI: 10.17323/2500-2597.2021.2.39.51
0 I © 2021 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.Org/licenses/by/4.0/).
Technology and strategic roadmaps have long been regarded as flexible tools that can support strategic and long-range planning by matching long-term goals with short-term actions and specific technology solutions [Farrukh et al., 2003]. Since its first introduction by Motorola in the 1980s, roadmapping has been applied to various contexts, including technology forecasting [Gerdsri, 2007], new product development [Petrick, Echols, 2004; Lee et al., 2008], service planning [Cho, Lee, 2014], and R&D project planning [Cho et al., 2016]. On the one hand, regardless of its context, the forward-looking feature of roadmapping helps organizations manage the fuzzy front-end of innovation and survive in turbulent environments, enabling them to reduce uncertainty by collecting the information required to plan for the future. On the other hand, it must be combined with other methods to be suitable for the context.
Among the approaches proposed for roadmapping, the two main streams are the data-driven and expert-based approaches. The data-driven approach uses hard data such as patents to investigate past trends to predict the future [Geum et al., 2015], whereas the expert-based approach relies heavily on expert insights - that is, soft data produced during roadmapping workshops [Phaal et al., 2004]. The representative research group of the former approach is the Seoul School, whereas that of the latter is the Cambridge School [Park et al., 2020]. These two approaches have distinct advantages and disadvantages. The data-driven approach, although it takes a retrospective perspective, utilizes the insights derived from reliable sources to implement systematic analysis techniques. Given recent advances in data analytics, such as natural language processing, deep learning, and artificial intelligence, as well as expanded data sources for investigating innovation activities, the potential for data to support human decision-making is considerable. However, the expert-based approach facilitates the use of tacit knowledge that is not available in the public domain. Furthermore, based on this knowledge, such an approach enables the setting of goals to achieve a desirable future outcome, supporting normative forecasting and exploratory forecasting. Accordingly, the two approaches can be complementary, and if they are implemented well together in the roadmapping process, they can support better decision-making related to science and technology innovation. Nevertheless, few previous attempts have been made to address this issue; the few exceptions include the work [Kostoff, Schaller, 2001], which mentions a hybrid method that combines a computer-based approach (considered a data-driven approach in this paper, emphasizing the importance of data) and an expert-based approach, and [Lee et al., 2007], which summarizes the data analysis techniques that can be used at each roadmapping stage. Indeed, the role of data in successful roadmapping has been highlighted
in previous studies [Lee et al., 2011; Schimpf, Abele, 2019]. The quantity and quality of information provided to support roadmapping can affect its results. To address this gap in the research, therefore, this study investigates the integration of data-driven approaches with expert insights during roadmapping. For this purpose, a workshop-based roadmapping method was combined with data-driven methods to test this approach in the context of technology planning for the automobile industry, Hyundai Motor Company. A single case study using an ethnographic approach1 was adopted to collect data on when, where, and how data analysis must be conducted to support experts' discussions. Thus, data needs during the roadmapping process were presented along with the techniques to visualize the data analysis results. The research findings open a discussion on how to integrate data-driven methods with expert insights during roadmapping based on the trade-offs between the two data types and suggest possible opportunities for the future development of roadmapping.
Literature Review
Roadmapping is defined as "A process that mobilizes structured systems thinking visual methods (e.g., road-map 'canvas' and participative approaches to address organizational challenges and opportunities), supporting communication and alignment for strategic planning and innovation management within and between organizations at the firm and sector levels" [Park et al., 2020, p. 2]. It helps organizations better prepare for technological change and offers a tool for corporate foresight [Linton, Walsh, 2004]. Indeed, organizations that use corporate foresight more often are more likely to be engaged in roadmapping and produce more innovation [Yoon et al., 2019]. Due to these advantages, roadmapping has gained significant attention recently and relevant research has increased notably in number [Carvalho et al., 2013; Park et al., 2020]. The number of roadmapping studies exceeded those on other popular planning tools such as Delphi, scenario planning and modeling/simulation [Park et al., 2020]. They also noted that the growth and prosperity of roadmapping studies have led to several research streams, with two distinguished ones including the focus on the design of roadmapping processes, outputs, and on the development of supportive tools for roadmapping [Park et al., 2020]. These research streams are related to roadmapping approaches that are classified into three categories: expert-based, computer-based, and hybrid approaches [Kostoff, Schaller, 2001]. The expert-based approach relies on insights in developing roadmaps and involves holding a series of workshops to identify roadmap elements and their relationships [Wells et al., 2004; Phaal, Muller, 2009; Farrukh et al., 2003; Phaal et al., 2007]. Moreover, cross-functional roadmapping teams can be organized to provide suf-
1 Ethnographic approach suggests non formalized, contextually adaptive gathering and analysis of empirical data.
ficient knowledge for successful roadmapping [Phaal et al., 2003; Gerdsri et al., 2010; Phaal et al., 2004]. The relevant studies mainly focus on the development of roadmapping processes or roadmap canvases and on identifying key success factors for procedure. As a result, numerous roadmapping methods have been proposed for choosing technology alternatives [Garcia, Bray, 1997], introducing scenario planning [Groenveld, 1997], supporting fast-starting roadmapping [Phaal et al., 2003], and managing emerging technologies [Gerdsri, 2007]. In addition to the roadmapping process, previous studies also proposed roadmap canvases, both as roadmapping outputs and as roadmapping guidelines. The most representative format is a time-based multi-layered chart, with the top layer mapping business trends and drivers, the middle layer mapping products/services/functions, and the bottom layer mapping technologies. However, these formats can change according to the purposes behind the roadmap-ping effort. Geum et al. [Geum et al., 2013] proposed a roadmap canvas specifically for open innovation, titled the dual-technology roadmap. Likewise, the relevant studies have investigated the design and customization of the roadmapping process, the structure of the road-map canvas, and the key success factors of roadmap-ping. Although expert-based roadmapping based on expert insights is appropriate for corporative foresight, its success may depend highly upon individuals' capabilities—that is, their ability to innovate, willingness to share information, and prior experiences used to justify their decisions. Therefore, it may not be effective in some cultures in which discussions are not encouraged or in areas of convergence in which expert insights are not sufficient to provide all necessary knowledge for roadmapping.
On the other hand, applying data analysis techniques that require a computer to create roadmaps has increased rapidly, possibly with advances in big-data analytics. Producing roadmaps solely via computer-based analysis, where roadmap elements and their relationships are identified without expert intervention, is called a computer-based approach. On the contrary, roadmapping that involves expert and computer-based analyses constitutes a hybrid approach. Previous studies adopting these approaches have generally proposed novel roadmapping methods (i.e., analysis methods) and tested them in practice. Accordingly, these approaches align with the 'application-and-proposition' research stream. First, computer-based roadmapping commonly employs patent data, which is regarded as one of the most rich and reliable sources of innovation. These roadmaps, sometimes called patent roadmaps, were developed to investigate technology trends [Jeong, Yoon, 2015; Jeong et al., 2015], monitor competitors [Lee et al., 2012; Yu, Zhang, 2019], or establish R&D strategies [Suh, Park, 2009]. Using patent data for road-mapping is advantageous because such data improves the credibility of roadmapping outputs. Nevertheless, patent roadmaps have shortcomings when used for corporate foresight due to its inherent retrospective
nature and its limited consideration of corporate-level strategies in developing roadmaps. Second, a hybrid approach aims to overcome the limitations of computer-based approaches and those of expert-based approaches to increase the results' objectivity while maintaining interactions among experts during roadmapping. Existing studies on hybrid roadmap-ping concentrated on tools to support decisions before, during, and/or after the process. Indeed, expert-based roadmapping often introduced strategy-making methods such as scenario planning and evaluation such as technology valuation to adopt information obtained from them. Many studies combined roadmapping with other decision-supporting tools emphasizing responses to uncertainty. For example, multiple scenarios and their impact on roadmaps are considered [Geum et al., 2014; Lee, Geum, 2017], the robustness of roadmaps is analyzed [Lee et al., 2016], and the impact of changes from external and internal factors on road-maps is evaluated to determine whether the roadmap needs revision [Gerdsri et al., 2019]. Other studies proposed a set of tools for various other purposes, such as a patent and portfolio analysis for prioritization [Lee et al., 2007] and a design-structure matrix for analyzing relationships among roadmap elements [Son et al., 2018]. These decision-supporting tools should greatly improve roadmapping performance, but few attempts have been made to understand how they are used in a real organizational setting. Our knowledge regarding the data source of hybrid roadmapping's needs, how it embeds into the overall organizational process, and how it evolved amongst rapid change is relatively limited. This study aligns with existing literature on hybrid roadmapping and tries to fill the research gap. Consequently, it highlights issues regarding a methodological aspect of hybrid roadmapping and its application at a large corporate organization [Park et al., 2020; Amati et al., 2020; Simonse et al., 2015]. Furthermore, we focus on the information sources necessary to ensure roadmaps' quality.
Proposed Hybrid Approach
The proposed approach has three stages, as presented in Figure 1. This process is based on Cambridge's S-Plan [Phaal et al., 2007], in that the first two stages correspond to S-Plan's landscaping and the last stage is related to the S-Plan's topic mapping. The first stage focuses on the ideation of innovation opportunities, which may come from short-term market and business needs or long-term changes in the technological and business environment. Thus, data for internal and external environment analyses are necessary. The ideas proposed in the first stage are then evaluated in the second stage. After the ideas are grouped into several topics, these topics are prioritized. Finally, for the topics selected in the second stage, detailed plans for pursuing each of them are established. Given that detailed planning requires a comprehensive review of relevant technologies, data analysis results that represent
technology trends and available technology solutions are needed at this stage. Using the workshop-based approach, we listed data sources that can support decision-making at each stage of roadmapping based on practical needs and we aimed to integrate these data-driven methods with expert insights.
Stage 1. Ideation
The first stage of roadmapping aims to identify innovation opportunities. The first roadmapping workshop is organized for this purpose. Workshop participants are encouraged to propose such opportunities by answering certain questions - why, what, and how. Those opportunities may arise from the technology-push and market-pull approaches. Technology push derives from new technologies seeking a useful application, whereas market-pull considers innovation opportunities based on market and business demands. Taking a longer perspective, it is necessary to address emerging changes in technological and business environments to facilitate insights that lead to disruptive innovation. During this discussion, although experts propose new business and technology opportunities based on their insights and data analysis, the provision of further information regarding internal and external environment analyses can promote their discussions. Internal sources of information, including customer complaints and survey results, help experts understand current market needs, as well as system, product, or service failures, to examine the limitations of current business offerings and past projects to identify actions taken to overcome the limitations of strategies at a higher level. However, external sources of information can be used to describe trends in terms of patents, publications, and media based on which emerging technologies and competitor's activities are investigated to identify opportunities. Particularly in cases of high uncertainty, future scenarios can be developed to derive various ideas for each scenario; innovation opportunities are captured
for the scenario. Figure 2 describes a workshop template for this stage. If j scenarios are built, j opportunity maps need to be created: some of innovation opportunities may be common across multiple scenarios, while others may be specific to a particular scenario. In the figure, topic (i,j) indicates the ith topic from the jth scenario. In order to help understand scenarios and seize innovation opportunities easily, the use of a value proposition canvas is recommended. The value proposition canvas is a graphical expression of what customers do, need, and suffer from in a specific context and further help design product and service offerings to satisfy the customers [Osterwalder et al., 2014].
Stage 2. Selection
Whereas the first stage is aimed at identifying various ideas, the second stage targets idea selection. As the ideas developed in the first stage may involve similar concepts, they must be grouped into several topics. Likewise, if common ideas are submitted from different scenarios, those ideas must be merged into a single topic. At this stage, many discussions among the workshop participants are held to define the scope and concept of each topic. Once the concept of each topic becomes clear, those topics are prioritized to select the most important ones. Generally, as promising topics that an organization is capable of implementing are considered valuable as business opportunities, two criteria - attractiveness and feasibility - are used for prioritization.
Then, a strategic technology roadmap is established on a selected topic, for which experts' panels are used. The experts use their insight to set a vision for the topic, discuss the current state of the topic, and establish the milestones to achieve the vision based on the current state. During this process, panels often face situations in which all information on the relevant technologies for the topic are not available, particularly in the era of uncertainty and convergence. Nevertheless, a compre-
Figure 1. Overall Roadmapping Process
Internal environment analysis (complaints, survey, system failure, past projects etc.) External environment analysis (scenarios, patents, publications, media etc.)
Identifying ideas to respond to market / business needs (short-term) Identifying ideas to prepare for changes in environment (long-term)
Prioritisation analysis (patents, publications, media etc.)
Grouping the ideas (product/service/technology) into topics Prioritise the topics (or technologies) based on evaluation criteria
Portfolio analysis (patents, publications, media etc.)
• Investigating and evaluating technologies from various perspectives
• Establishing detailed plans for technology acquisition (M/B/C)]
Source: [Phaal et al., 2007].
Figure 2. Opportunity Map (adapted from the strategic landscape of S-Plan)
Customer job
Gains Pains
Gain creators
Pain relievers
Innovation opportunities
Value Proposition Canvas
Note: Topic (i, j) means i-th topic of j-th scenario. Source: compiled by the authors.
Scenario j
Trends Drivers Policies
Products Service Systems Functions
Technologies
Facilitators Challenges
Why
What
Topic i
How
Opportunity Map
2020
2022
2025
2027
Vision
hensive understanding of the corresponding technology trends and potential competitors is required for robust planning. Here, patent analyses discover available technology solutions within the target area as well as other areas. In particular, the recent advances in data analytics and visualization allow one to extract useful technical information from a large number of patent documents effectively. Emerging technological trends can be identified in the form of keywords or key concepts from the collection and analysis of topic-related patent documents. On the other hand, patent documents published in other areas can also be analyzed to summarize their trends to be referenced or converged, which supports the discussion during roadmapping. Here, the level of specificity for a strategic technology roadmap may vary by resource constraints and the roadmap's purpose. Figure 3 represents workshop templates for Stage 2. The map on the left is used for detailed planning for each topic, while the map on the right is for an aggregated level planning for all topics of concern.
Stage 3. Planning
The final stage is aimed at developing a detailed plan to pursue high-priority topics, particularly those focusing on technology planning. The technologies relevant to the topics are evaluated by four selection criteria: importance, urgency, development risk, and technological capabilities. Importance evaluates the criticality level of the organization's acquiring the technology. If the technology is likely to have a strong, positive impact on the organization and aligns with the organizational strategy, it will have a higher value. Urgency measures how immediately the technology is needed at the organization. Risk evaluates the degree of risk associated with the technology; if the technology requires complex technologies and high costs, its development risk will be high. Capability indicates the level of technology-related knowledge or expertise within the organization. These criteria can be adjusted to roadmapping contexts. Sub-criteria can
be designed, where decision-supporting techniques such as scoring models, the analytic hierarchical processes, or the analytical network processes are used to synthesize experts' evaluation results. Furthermore, data analysis can also support experts' evaluation at this stage. For example, patent analysis can serve as a reference for technological capabilities and impact analysis can serve as a reference for importance. Based on the analysis results, two portfolios are proposed: one to prioritize the items and the other to establish an action plan (see Figure 4).
Case Study Background
The three-stage workshop-based roadmapping process was performed in collaboration with Hyundai Motor Company, a South Korean automobile company. Recently, the automobile industry has encountered considerable challenges related to a dramatically changing business landscape caused by the emergence of the sharing economy, pressures caused by environmentally friendly automobiles, and the opportunities available for various forms of personal mobility. These unpredictable factors make the industry risky for incumbents but easy to enter for newcomers. Organizations in the industry are introducing roadmapping aggressively to search for new business and technology opportunities and they are investing in their R&D to cope with the expected changes in their industry Accordingly, the automobile industry was suitable for a case study, which was conducted over the course of two months (February and March 2018). The research team played the role of a roadmapping team that designed a process, recruited participants, facilitated the process, provided relevant information, and summarized the roadmapping results. During the process, we observed when and where information needs occurred, what kind of information was required, and how the data needed to be analyzed to support expert's decisions effectively during roadmapping.
Figure 3. Strategic Technology Roadmap (adapted from the topic map of S-Plan)
Topic 1
Trends Drivers Policies
Products Service Systems Functions
Technologies
Facilitators Challenges
Source: compiled by the authors.
Topic 4 _!_r
Topic 1 _r
Topic 2 _r
Topic 3 _I_r
Now Short-term Mid-term Long-term
Roadmapping Process
The three-stage roadmapping was implemented using two workshops along with tasks imposed between the workshops. The first and second stages were covered in the first workshop, while the third stage was conducted in the second workshop. The participating team was in charge of technology development related to the noise, vibration, and harshness (NVH) of automobiles and aimed to establish a long-term R&D plan for about 10 years given the new role of NVH technology in future mobility services: in general, a ten-year time horizon is considered to be appropriate for many organizations [Phaal, Muller, 2009]. All team members were involved in the roadmapping as a taskforce team. The roadmap-ping process started from process design, followed by the development of scenarios, identification of opportunities, and the development of strategies along with action plans. Here, short-term opportunities were searched for in the trend analysis of patents and publications, while long-term opportunities were identified from expert discussions. From the requirements of the participating team, the main target for data analysis
was set to patents and publications within and outside of the sector. As a result, a standardized process proposed in this study was customized as shown in Table 1. The first workshop was carried out on February 9, 2018 and was aimed at identifying new business and technology opportunities (Stage 1). Before the workshop, the team requested that future mobility scenarios be prepared because all the team members were engineers and, although NVH technologies can be expected to be influenced significantly by external factors, they needed sufficient time to think about those factors. Accordingly, five scenarios were proposed using three criteria - vehicle control, vehicle ownership, and new vehicles (see Figure 5). Then, to facilitate discussion, customer profiles that described the activities that customers were involved in while using the vehicles were developed for each scenario as well as a value proposition map that investigated the needs and wants expected during the activities. Finally, the participants generated new business and technology opportunities and explained them in terms of why the opportunity is needed, what the opportunities are, and how the op-
Figure 4. Portfolio Map for (a) Prioritization (left) and (b) Action Plan (right)
High
£ 3
Low
High
Outward transfer Internal R&D л; ig с 'С о Second-top priority
2 с о S
Withdraw R&D collaboration Low Top-priority
Low
Source: compiled by the authors.
Importance
High
Low
Importance
High
Table 1. Customized Roadmapping Process
Workshop Stage Experts' insights Data-driven methods (data sources)
1st round Ideation • Value proposition for each scenario • Innovation opportunities in each scenario • Scenario analysis by identifying trends (public and private reports)
Selection • Topic definition based on the opportunities • Topic evaluation using two criteria -attractiveness and feasibility • Technology trend analysis on the target area (patents and publications data) • Technology trend analysis on the reference area (publications data)
2nd round Planning • Technology definition for each topic • Technology evaluation using four criteria -urgency, risk, importance, and capability • Statistics and portfolio analysis on the evaluation results (evaluation data) • Potential collaboration partner analysis (publications data)
Source: authors.
portunity can be pursued (that is, the technologies required to do so).
These opportunities were discussed among participants and similar opportunities were merged, leading to 18 topics (Stage 2). A further evaluation of the attractiveness and feasibility of the 18 topics led to the selection of 11 topics along with 46 corresponding technologies for detailed planning. Of these, five topics were associated with multiple scenarios and were called G-topics (general-topics), whereas six were scenario-specific and were called C-topics (context-topics). Given the limited number of topics, internal discussions were carried out to choose the topics and technologies for further investigation.
On the other hand, trend analysis results based on patents and publications were provided to the participants for the identification of available technologies within and outside of the sector (see Figure 6). For this purpose, 27,411 publications were collected from the Scopus database. In total, 5,988 patents from the USPTO, 1,181 patents from the EPO, and 329 patents from the KPO were collected on automobile NVH technologies; these were published between January 2016 and
March 2018. To summarize the contents of the patents and publications effectively, topic modeling based on LDA2 was performed. This resulted in nine topics (29 subtopics) from patents and publications. The relationships between the topics proposed during the workshop and the topics obtained from data analysis were investigated by the research team and proposed to the roadmapping participants for reference. In addition, for each topic, the relevant keywords, the number of relevant documents, the major organizations, and key documents were summarized, as shown in Table 2. We also highlighted hot topics that are gaining more attention and cold topics that are losing attractiveness, which we defined based on the increasing rate of relevant documents, along with outlier patents defined as unique patents in terms of their technological content on the premise that those patents could have the potential to be disruptive technologies. This data analysis process helped the participants understand what opportunities are in line with the main technology trends and, more specifically, obtain a list of patents and publications worth reviewing while investigating each topic.
Figure 5. Future Mobility Scenarios
The driverless revolution fl A new age of accessible autonomy k .
o Incremental change o A world of car-sharing
Baseline scenario
Individual Shared
Vehicle ownership
A world with whole new type of mobility
• Vertical take-off and landing
• Flying passenger drone
• Hyperloop
• Wig craft ship
Extended mobility
New vehilcles
Value proposition Customer profiles
• Products and services • Customer jobs
• Gain creators • Gains
• Pain relievers • Pains
Scenario analysis
Source: authors.
2 Latent Dirichlet Allocation.
Table 2. Partial Topic Analysis Results (door locking apparatus)
Topic Door locking apparatus (US-topic5)
Keywords Position, actuator, locking, movable, lever, movement, lock, positions, latch, move
Document number 589
Key documents • One motor latch assembly with power cinch and power release having soft opening function (US20170089103A1) • Apparatus and method for actuating a switch or sensor (US20160230427A1) • Twist latch for compartment door (US20170218667A1) • Door lock device for vehicle (US20160340937A1) • Cinching latch assembly for vehicle (US20170306661A1)
Source: authors.
Furthermore, the participants suggested that recent psychoacoustic technologies could be applied to improve automobile NVH system, particularly for G-Topic 1. However, the topics were outside their areas of expertise, so another round of data analysis was necessary. Focusing on the psychoacoustic technologies, we collected 1,534 publications published from January 2016 to March 2018 from the Scopus database. Again, LDA-based topic modeling was used, resulting in ten topics (40 subtopics). For this study, we focused only on the publication data because the company was seeking collaboration partners in academia, which was the main source of the publications. Again, the research team matched relationships between the 13 technologies for G-topic 1 and ten topics identified during data analysis to help the roadmapping participants introduce emerging psychoacoustic technologies in the NVH system. Furthermore, these patent and publication analyses enabled the participating team to identify potential collaborators for pursuing the topics.
The second workshop, which targeted more focused discussions on the selected topics and relevant technologies, was held on March 29, 2018 (Stage 3). Between the first and second workshops, we asked five key participants to evaluate the 46 technologies associated with the 11 topics using four criteria - urgency, risk, importance, and capability - as well as precedent relationships between the technologies via a technology cross-impact matrix. This task encouraged the participants to think deeply about the topics and technologies and let them search for those technologies individually. This was expected to support discussions in the second workshop. Given the broad scope of the topics, we allowed the participants to leave some questions partly unanswered if they lacked expertise on the technology or failed to collect relevant information. On acquiring the results, the basic statistics of the technology - mean and standard deviation - and technological relationships were used to develop a preliminary technology roadmap for the second workshop, as shown in Figure 8.
In this workshop, participants presented and discussed various viewpoints, particularly in terms of the technologies with high standard deviations in their evaluation results. An in-depth discussion of the technologies with diverse viewpoints led the participants to share their ideas and reach a consensus. In addition, a first draft roadmap developed based on the evaluation results was presented to be modified according to the discussion results. This preliminary roadmap could have been developed for each topic or for all topics at an aggregated level. As the roadmapping was conducted at the team level, not the organizational level, we put all the topics and relevant technologies onto a single roadmap, as shown in Figure 9. Here, it should be noted that five C-topics were merged into a single topic due to the small number of relevant technologies for each topic. Accordingly, the roadmap included seven topic layers with 46 technologies. FFirst, the technologies were positioned according to their urgency values
SCOPUS publications DB
Figure 6. Topic Analysis Results for within the Sector
Topics from data analysis results (Ä,..., J)
USPTO patents DB
EPO patents DB
KPO patents DB
O-topic 1
Jy 13 O-topic 1
S lîg « .5 is O-topic 1
£ -a O-topic 1
—S S - CU y ^ s O-topic 1
TD 'Ü eu O 3 < ics from O-topic 1 K-topic 1
is K-topic 1
G £ £ K-topic 1
K-topic 1
K-topic 1
.........
Source: authors.
Figure 7. Topic Analysis Results for Outside the Sector
Topics from data analysis results on psycho-acoustic technologies
SCOPUS publications DB -1.534 Unstructured text data analysis DA topic modelli - s Bl'l| P2 I P3 I P4 I P5 I P6 I P7 I P8 I P9 I P10|
(L ng) Technobeies from workshop K* results on G-Topic 1 WJ* G1-1 G1-2 G1-3 G1-4 G1-5 G1-6 G1-7 • • • • • • • • • • • • • • • • • • • • • •
Source: authors. G1-8 G1-9 G1-10 G1-11 G1-12 G1-13
Figure 8. Technology Evaluation Results
а) Basic statistics on the evaluation results (partial)
Urgency Development risk Importance Technological capabilities
Mean Std Rank Mean Std Rank Mean Std Rank Mean Std Rank
2.75 1.5 24 4 0 2 3.8 0.447213595 9 2.75 1.483239607 7
2 1 37 3.6 0.547722558 6 3.6 0.547722558 15 1.4 0.547722558 40
3 1 19 2.4 0.894427191 35 3.4 0.547722558 25 1.75 1.140175425 30
3.4 1.140175425 10 3.6 0.894427191 6 3.6 0.894427191 15 1.2 0.447213595 45
3.6 0.894427191 8 3 1 21 3.8 0.447213595 9 1.6 1.341640786 36
Source: authors.
Figure 9. Technology Roadmap
Source: authors.
Table 3. Аggregated Evaluation Results at the Topic Level
Topic Urgency Risk Importance Capabilities
Mean Std Rank Mean Std Rank Mean Std Rank Mean Std Rank
O-topic 1 2.74 0.52 10 2.95 0.30 7 3.30 0.22 7 2.01 0.67 7
O-topic 2 2.75 0.68 9 3.60 0.68 1 2.95 0.54 9 1.75 0.85 9
O-topic 3 3.15 0.89 5 2.10 0.52 11 3.55 0.62 4 2.40 0.68 2
O-topic 4 3.26 0.49 3 2.78 0.43 8 3.79 0.43 3 2.15 0.62 6
O-topic 5 2.80 0.57 6 3.20 0.76 3 2.90 1.24 10 2.25 0.87 4
O-topic 6 2.77 0.91 8 3.00 0.91 6 2.73 0.89 11 2.27 0.98 3
K-topic 1 3.40 1.52 2 3.40 1.82 2 4.20 0.45 1 2.00 1.00 8
K-topic 2 3.20 1.10 4 2.20 1.10 10 4.00 1.22 2 3.40 1.67 1
K-topic 3 3.80 1.30 1 2.50 1.73 9 3.40 1.14 6 2.25 1.26 4
K-topic 4 2.80 1.30 6 3.20 1.48 3 3.20 1.10 8 1.25 0.50 11
K-topic 5 2.17 1.48 11 3.20 1.04 3 3.53 0.49 5 1.67 0.53 10
Source: authors.
and the links were established based on the values in the cross-impact matrix, where their importance values were presented as reference information. The participants reviewed the first-cut roadmap and adjusted values and positions based upon mutual agreement. Adding or deleting more technologies was allowed at this stage, although it did not occur in this case study. Finally, the technology evaluation results were aggregated by topic (see Table 3), based upon which portfolio map types were developed - one for prioritization and another for action plans as shown in Figure 10. With the average index value to separate high and low space in the map, five out of 11 topics (G-Topic 1, G-Topic 3, G-Topic 4, C-Topic 2, and C-Topic 3) were positioned in the fourth quadrant of the prioritization map, signifying top priorities of development. Among these five topics, four (G-Topic 1, G-Topic 3, G-Topic 4, and C-Topic 3) were located in the fourth quadrant, while only C-Topic 2 was in the first quadrant of the action plan map. Thus, an R&D collaboration strategy was recommended for
the first four topics, producing an industry-academic collaboration lab funded by Hyundai Motor Company. Internal R&D was designed for C-Topic 2. When investigating the relationships between short-term emerging topics identified from the patent and publication data analysis and long-term promising topics identified from expert insights, we found that improving the internal capabilities in E, F, and I, which are associated with the five top-priority topics (G-Top-ic 1, G-Topic 3, G-Topic 4, C-Topic 2, and C-Topic 3), would be greatly helpful in preparing for the future and gaining competitive advantages. Furthermore, we also found that G-Topic 1 could benefit significantly from recent developments in psychoacoustic technologies, since the topic is characterized as a technology convergence between NVH and psychoacoustic technologies. Therefore, a link between technologies for G-Topic 1 and the topic analysis results for psychoacoustic technologies were considered to identify potential collaboration partners.
Figure 10. Portfolio Maps
4
Second-top priority
Or-ropc '4 O.Topic- 1
• ' . ............................'' .■a 3
Monitoring Topic 1 O-Topit l"\ O-Topic 3 ap <3
• O-Topic 2'' O-Topic 3 2
...........Top-priority
Internal R&D .*' K-Topic 2
O-Topic.6
Monitoring
K-Topic 3 R&D Collaboration
O-Topic 3
• O-Topic 4 O-Topic 5 9 • # O-Tpic 1 K-Topic 1
O-Topic 2 $
K-Topic 4
Importance
K-Topic 5
3
Importance
Source: authors.
2
2
4
2
Discussions
Recommended situations for applying the method
With the emergence of disruptive technologies and innovative business models, companies are challenged by ongoing technological revolutions and rapidly changing business conditions, leading to an increasing level of uncertainty. The proposed approach that integrates data-driven methods with expert insights is useful particularly for technology planning under such uncertainty. Expert insights are key to long-term planning by setting a vision and strategy for the organization, while data-driven methods enable users to understand and prioritize innovation opportunities made available by recent disruptive technologies. In addition, the data-driven methods help one to collect and synthesize expert insights systematically. Under uncertainty, experts may have different subjective opinions on the same candidate opportunities, possibly due to their different experiences and knowledge of the target technological domain. Thus, it is essential to integrate their knowledge for reliable decision-making, where data-driven methods provide objective information around candidate opportunities.
Bias
Despite the value of combining expert insights with data-driven methods, using several data-based methods is not always advantageous because data collection and analysis usually requires a huge amount of time and effort, thus delaying the roadmapping process. On the other hand, agile roadmapping is valuable under uncertainty, capturing value in short-term initiatives. Accordingly, understanding the needs for data analysis is indispensable for designing the roadmapping process along with accessing the right data and having the most qualified people as roadmapping participants. Notably, said participants may depend too much upon data analysis results, thus limiting their creativity in developing new ideas. Hence, an optimal use of data analysis results for effective roadmapping must be determined. Furthermore, additional data sources necessitate a process for selecting reliable ones, constructing a balanced portfolio, and developing effective analysis methods — issues worth addressing.
Implications for theory
While most existing studies on roadmapping focused either on workshop-based or data-driven approaches [Park et al., 2020], this study emphasized the integrated use of expert insights and data-driven methods. On the one hand, opportunity capture by eliciting knowledge from experts is important since it involves tacit knowledge sharing. On the other hand, opportunity capture by analyzing technological and market data is also significant given that it enables one to generate, identify, and evaluate more ideas. How to best combine the two types of knowledge - one from expert insights and the other from data analysis - can be an important topic for roadmapping research.
Implications for practice
The research findings offered several implications for practice in the field of roadmapping. First, data analysis needs largely occurred in three areas: understanding research trends within the sector, identifying available technologies in another relevant sector, and collecting and summarizing expert opinions. With today's increased technological complexity and emerging breakthrough and/or converging technologies, data analysis is essential to improving roadmap quality. Furthermore, offering data analysis results may enhance communication among experts. By providing objective information, a data analysis prevents a single person, usually a senior manager, from dominating the discussion. Future research is needed to examine how the data analysis results can be used to enhance road-mapping quality.
Application notes
Before initiating a roadmapping process, it is necessary to clearly define the scope and purpose of such an endeavor including a plan of how the outputs will be used. Designing a roadmapping process within a limited budget is also critical for successful roadmapping, particularly for a hybrid method where both experts and data-driven approaches are used. Regarding the experts, the most motivated and qualified people need to be identified to be engaged in roadmapping. Using appropriate templates can help to elicit and share expert knowledge. As to the data, the data analysis results should be able to fill in the knowledge gaps of the experts. Therefore, the data analysis needs, procedures, and results are recommended to be carefully planned. Otherwise, the analysis results cannot be integrated into workshop-based roadmapping. The cost and benefit of analyzing each type of data should be considered as well to optimize the use of data-driven methods.
Lessons from the case study
First, expert insights produced creative ideas with information about relevant projects (past and ongoing) and about learning from them, whereas data analysis results mostly offered feasible ideas regarding competitor trends and available technologies in other sectors. These two approaches are complementary, but analysis results must be provided in a correct form at the correct level of detail for data analysis results to be useful for experts. Recent data analysis and visualization techniques can be introduced to effectively use data. Second, most computer-based roadmapping literature mainly considered two data sources as targets for analysis: patents and publications, which are useful in analyzing past trends. However, more data sources that present competitors' plans (e.g., news and You-Tube) and/or expert insights from outside of organizations (e.g., LinkedIn and podcasts) will be available, going beyond traditional analyses of patents and publications. These data sources can be linked to generate more valuable implications.
Areas for future research
The workshop observation raised several research gaps to address, suggesting future research directions. First, in the case study, the data analysis needs stem from roadmapping at a team level, and different needs could be addressed via roadmapping at the organizational level. The needs may also vary by roadmapping context, such as long-term versus short-term planning, product versus service development, defining versus solving problems, and internal R&D versus collaboration strategies. According to [Schimpf, Abele, 2019], data sources useful for roadmapping include external road-maps, associations, market analyses, suppliers, customers, users, research organizations and universities, competitors, consulting companies, legislation, journals, and media. Useful public and private data sources for context-specific roadmapping must be studied further. Second, continuous research is needed to develop hybrid roadmapping processes and methodolo-gies.3 Combined with newly available data sources and methods, the roadmapping process can become more efficient. Moreover, a hybrid process can also benefit significantly from introducing roadmapping systems, which enables systematic data collection, sharing, and elaboration across organizational units [Amati et al., 2020]. It enables the integrated application of those methods and easy updates of roadmaps [Phaal et al., 2004; Lee, Park, 2005]. Thus, future research is necessary to create a framework for developing, evaluating, and improving a hybrid roadmapping process. Finally, future research must address challenges by the timely updating of roadmaps to keep them alive. Changing business environments and emerging promising technologies can be monitored continuously via data collection and analysis. How to evaluate a roadmap's status and how to choose when to revise it need further investigation.
Conclusions
This study discussed how to combine data-driven methods with expert insights during roadmapping. Today's business environment is characterized as being full of uncertainties. On the one hand, the rapid emergence of disruptive technologies4 has changed the way organizations do business. On the other hand, social pressures on technologies or other unpredictable social changes such as the present pandemic also make it hard for an organization to plan the future. Accordingly, a workshop-based approach can help an organization to quickly respond to the changing environment, enabling agile roadmapping, and allowing organizations to also establish long-term planning based
on expert insights. Here, an appropriate use of data is expected to supplement limited knowledge during the expert-driven roadmapping.
In this study, a workshop-based roadmap aiming for 10-year technology planning to deal with uncertainties was implemented in collaboration with Hyundai Motor Company. We focused on the data analysis needs of the workshop participants and the way the analysis results were used in participants' decision-making. The research findings indicated that discussions among participants were facilitated by the use of data analysis results: they were useful in having a forward-looking perspective for the ideation stage and collecting diverse opinions to develop a first-cut roadmap for the selection stage. For the planning stage, data were useful for understanding emerging trends within the sector and identifying available technologies and collaboration partners outside of the sector. Consequently, this study contributes to the advances in roadmapping methodologies by suggesting the necessity of a hybrid approach that combines data analysis results and expert insights. Furthermore, this is one of the few studies on corporate roadmap-ping in a real setting, and hence, it can have practical contributions as well.
Despite its meaningful contributions, this study has several limitations. First, only a single case study was conducted in each of the contexts of 1) team-level roadmapping; 2) the automobile industry; and 3) the Asian context. The level of hard data required for robust roadmapping may change due to various factors such as the purpose of a given roadmapping, industry condition and organizational and possibly national culture. More cases are needed to improve the external validity of the research findings. Second, most of the findings were derived from observations on the roadmapping process itself. Although an ethnographical approach was adopted in this study, interviews with participants or surveys on data analysis needs during roadmapping across organizations could produce more meaningful insights. Accordingly, further analysis on the before-and after-roadmapping stages can be conducted to understand data needs and satisfaction. Finally, this study discussed how to integrate data-driven methods with expert insights but failed to propose a new roadmap-ping framework to integrate a data-driven approach with expert insights. Recent advances in data analysis techniques along with the emergence of numerous data sources are expected to provide various implications to support expert ideation and decision-making during roadmapping. Future research will continue to address those issues.
3 Various methods have been applied to support roadmapping, including technology radar, portfolios, creativity methods, strategy maps, balanced scorecards, scenario analyses, quality function development, technology maps, maturity models, regression, and Delphi studies [Schimpf, Abele, 2019].
4 Among them are self-driving cars, robots, artificial intelligence, big data, the Internet of Things, mobile technology, virtual reality, blockchain, FinTech, drones, 3D printing, digital healthcare, bio-healthcare, and new materials and energies.
References
Amati G., Motta V., Vecchiato R. (202o) Roadmapping for innovation management: Evidence from Pirelli. R&D Management, S0(4), 462477. https://doi.org/l0.llll/radm.l2398 Carvalho M.M., Fleury A., Lopes A.P. (20l3) An overview of the literature on technology roadmapping (TRM): Contributions and trends.
Technological Forecasting and Social Change, 80(7), l4l8-l437. https://doi.org/10.1016/j.techfore.2012.11.008 Cho C., Lee S. (2014) Strategic planning using service roadmaps. The Service Industries Journal, 34(12), 999-1020. https://doi.org/l0.l080/ 02642069.2014.91S9S1
Cho Y., Yoon S.P., Kim K.S. (201б) An industrial technology roadmap for supporting public R&D planning. Technological Forecasting and
Social Change, 107, 1-12. https://doi.org/10.1016/j.techfore.2016.03.006 Farrukh C., Phaal R., Probert D. (2003) Technology roadmapping: Linking technology resources into business planning. International
Journal of Technology Management, 26(1), 2-19. 10.1S04/IJTM.2003.003140 Garcia M.L., Bray O.H. (1997) Fundamentals of technology roadmapping, Albuquerque, NM: Sandia National Labs.
Gerdsri N. (2007) An analytical approach to building a technology development envelope (TDE) for roadmapping of emerging technologies.
International Journal of Innovation and Technology Management, 4(02), 121-13S. https://doi.org/l0.ll42/S02l987700700l004 Gerdsri N., Assakul P., Vatananan R.S. (2010) An activity guideline for technology roadmapping implementation. Technology Analysis &
Strategic Management, 22(2), 229-242. https://doi.org/l0.l080/09S37320903498SS3 Gerdsri N., Puengrusme S., Vatananan R., Tansurat P. (2019) Conceptual framework to assess the impacts of changes on the status of a
roadmap. Journal of Engineering and Technology Management, S2, 16-31. https://doi.org/l0.l0l6/j.jengtecman.20l7.l2.00l Geum Y., Kim J., Son C., Park Y. (2013) Development of dual technology roadmap (TRM) for open innovation: Structure and typology.
Journal of Engineering and Technology Management, 30(3), 309-32S. https://doi.org/l0.l0l6/j.jengtecman.20l3.06.00l Geum Y., Lee H., Lee Y., Park Y. (20lS) Development of data-driven technology roadmap considering dependency: An ARM-based
technology roadmapping. Technological Forecasting and Social Change, 91, 264-279. https://doi.org/l0.l0l6/j.techfore.20l4.03.003 Geum Y., Lee S., Park Y. (2014) Combining technology roadmap and system dynamics simulation to support scenario-planning: A case of
car-sharing service. Computers & Industrial Engineering, 71, 37-49. https://doi.org/l0.l0l6/jxie.20l4.02.007 Groenveld P. (1997) Roadmapping integrates business and technology. Research Technology Management, 40(S), 48-SS. https://doi.org/l0. 1080/089S6308.1997.116711S7
Jeong Y., Lee K., Yoon B., Phaal R. (201S) Development of a patent roadmap through the Generative Topographic Mapping and Bass
diffusion model. Journal of Engineering and Technology Management, 38, S3-70. https://doi.org/l0.l0l6/j.jengtecman.20lS.08.006 Jeong Y., Yoon B. (201S) Development of patent roadmap based on technology roadmap by analyzing patterns of patent development.
Technovation, 39, 37-S2. https://doi.org/l0.l0l6/j.technovation.20l4.03.00l Kostoff R.N., Schaller R.R. (2001) Science and technology roadmaps. IEEE Transactions on Engineering Management, 48(2), 132-143. DOI:l0.ll09/l7.922473
Lee C., Kim J., Lee S. (2016) Towards robust technology roadmapping: How to diagnose the vulnerability oforganisational plans. Technological
Forecasting and Social Change, 111, 164-17S. https://doi.org/l0.l0l6/j.techfore.20l6.06.022 Lee H., Geum Y. (2017) Development of the scenario-based technology roadmap considering layer heterogeneity: An approach using CIA
and AHP. Technological Forecasting and Social Change, 117, 12-24. https://doi.org/l0.l0l6/j.techfore.20l7.0l.0l6 Lee J.H., Kim H.I., Phaal R. (2012) An analysis of factors improving technology roadmap credibility: A communications theory assessment
of roadmapping processes. Technological Forecasting and Social Change, 79(2), 263-280. https://doi.org/l0.l0l6/j.techfore.20ll.0S.003 Lee J.H., Phaal R., Lee C. (2011) An empirical analysis of the determinants of technology roadmap utilization. R&D Management, 41(S),
48S-S08. https://doi.org/l0.llll/j.l467-93l0.20ll.006S7.x Lee S., Kang S., Park Y., Park Y. (2007) Technology roadmapping for R&D planning: The case of the Korean parts and materials industry.
Technovation, 27(8), 433-44S. https://doi.org/l0.l0l6/j.technovation.2007.02.0ll Lee S., Lee S., Seol H., Park Y. (2008) Using patent information for designing new product and technology: Keyword based technology
roadmapping. R&D Management, 38(2), 169-188. https://doi.org/l0.llll/j.l467-93l0.2008.00S09.x Lee S., Park Y. (200S) Customization of technology roadmaps according to roadmapping purposes: Overall process and detailed modules.
Technological Forecasting and Social Change, 72(S), S67-S83. https://doi.org/l0.l0l6/j.techfore.2004.ll.006 Lee S., Yoon B., Lee C., Park J. (2009) Business planning based on technological capabilities: Patent analysis for technology-driven
roadmapping. Technological Forecasting and Social Change, 76(6), 769-786. https://doi.org/l0.l0l6/j.techfore.2009.0l.003 Linton J.D., Walsh S.T. (2004) Integrating innovation and learning curve theory: an enabler for moving nanotechnologies and other emerging
process technologies into production. R&D Management, 34(S), S17-S26. https://doi.org/l0.llll/j.l467-93l0.2004.003S9.x Osterwalder A., Pigneur Y., Bernarda G., Smith A. (2014) Value Proposition Design: How to Create Products and Services Customers Want, New York: John Wiley & Sons.
Park H., Phaal R., Ho J.Y., O'Sullivan E. (2020) Twenty years of technology and strategic roadmapping research: A school of thought
perspective. Technological Forecasting and Social Change, 1S4, 11996S. https://doi.org/l0.l0l6/j.techfore.2020.ll996S Petrick I.J., Echols A.E. (2004) Technology roadmapping in review: A tool for making sustainable new product development decisions.
Technological Forecasting and Social Change, 71(1-2), 81-100. https://doi.org/l0.l0l6/S0040-l62S(03)00064-7 Phaal R., Farrukh C., Mitchell R., Probert D. (2003) Starting-up roadmapping fast. Research-Technology Management, 46(2), S2-S9. https://
doi.org/l0.l080/089S6308.2003.ll67lSSS Phaal R., Farrukh C.J., Probert D.R. (2004) Technology roadmapping — A planning framework for evolution and revolution. Technological
Forecasting and Social Change, 71(1-2), S-26. https://doi.org/l0.l0l6/S0040-l62S(03)00072-6 Phaal R., Farrukh C.J., Probert D.R. (2007) Strategic roadmapping: A workshop-based approach for identifying and exploring strategic
issues and opportunities. Engineering Management Journal, 19(1), 3-12. https://doi.org/l0.l080/l0429247.2007.ll43l7l6 Phaal R., Muller G. (2009) An architectural framework for roadmapping: Towards visual strategy. Technological Forecasting and Social
Change, 76(1), 39-49. https://doi.org/l0.l0l6/j.techfore.2008.03.0l8 Schimpf S., Abele T. (2019) How German companies apply roadmapping: Evidence from an empirical study. Journal of Engineering and
Technology Management, S2, 74-88. https://doi.org/l0.l0l6/j.jengtecman.20l7.l0.00l Simonse L.W., Hultink E.J., Buijs J.A. (201S) Innovation roadmapping: Building concepts from practitioners' insights. Journal of Product
Innovation Management, 32(6), 904-924. https://doi.org/l0.llll/jpim.l2208 Son H., Kwon Y., Park S.C., Lee S. (2018) Using a design structure matrix to support technology roadmapping for product-service systems.
Technology Analysis & Strategic Management, 30(3), 337-3S0. https://doi.org/l0.l080/09S3732S.20l7.l3l0377 Suh J.H., Park S.C. (2009) Service-oriented technology roadmap (SoTRM) using patent map for R&D strategy of service industry. Expert
Systems With Applications, 36(3), 67S4-6772. https://doi.org/l0.l0l6/j.eswa.2008.08.062 Wells R., Phaal R., Farrukh C., Probert D. (2004) Technology roadmapping for a service organization. Research-Technology Management,
47(2), 46-S1. https://doi.org/l0.l080/089S6308.2004.ll67l6l9 Yoon J., Kim Y.J., Vonortas N.S., Han S.W. (2019) A moderated mediation model of technology roadmapping and innovation: The roles of corporate foresight and organizational support. Journal of Engineering and Technology Management, S2, 61-73. https://doi.org/l0.l0l6/j. jengtecman.20l7.l0.002
Yu X., Zhang B. (2019) Obtaining advantages from technology revolution: A patent roadmap for competition analysis and strategy planning. Technological Forecasting and Social Change, 14S, 273-283. https://doi.org/l0.l0l6/j.techfore.20l7.l0.008
