CC BY-NC-ND
215
Technology Selection Using the TOPSIS Method
Katarzyna Halicka
Professor, [email protected] Bialystok University of Technology, 45A, Wiejska Street, 15-351 Bialystok, Poland
Abstract
Innovative technologies are increasingly determining the competitive advantage of enterprises. They also form the basis for modern manufacturing processes, enabling them to meet the needs of society. Awareness of the need for technological development has become widespread, which has been confirmed by international and national programs, scientific and research activities, as well as emerging institutions. Considering the increasing demand for innovative technologies and a developed market, it appears important to use specific methods and tools for the effective analysis and selection of technologies. This paper presents a proposal to use multi-attribute
decision-making methods during technology assessment and selection. The proposed concept combines an S-life-cycle analysis (S-LCA), which determines the performance of a technology, the method of Technology Readiness Levels (TRL), which examines the technological maturity, and the TOPSIS method, which allows for developing a technology ranking. To verify this approach, the example of a ranking and selection of the best road technology in Poland is presented, considering the proposed set of criteria and sub-criteria. In the assessment, the criteria for innovation, competitiveness, and usefulness of this technology were used in addition to S-LSA and TRL methods.
Keywords: technology; innovation; technology selection; technology assessment; technology readiness levels; TOPSIS; Multi-Attribute Decision-Making methods
Citation: Halicka K. (2020) Technology Selection Using the TOPSIS Method. Foresight and STI Governance, vol. 14, no 1, pp. 85-96. DOI: 10.17323/2500-2597.2020.1.85.96
0 I © 2020 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.Org/licenses/by/4.0/).
The selection of technologies is based on a set of pre-defined criteria, the aggregation of which allows one to create a ranking. The task of the ranking is to collect information on alternative technologies from different sources and to assess these alternatives based on a set of criteria, considering the priorities of the organization that carries out or commissions the assessment. The selection of appropriate technologies allows for the empirical evaluation of current technology parameters and its development potential. It is used for the assessment of a set of existing technical solutions, a portfolio of technologies, products, or patented inventions owned by an organization. Selection and ranking require either the measurement of relevant parameters of the technology, an organization and its market environment, or the use of expert evaluations to determine the values of some of these parameters. Technology Assessment (TA) is an integral part of the ranking.
The concept of technology assessment was first introduced in the mid-1960s to determine the consequences arising from the development of new technologies [van den Ende et al., 1998; Carlsen et al., 2010]. Over time, the concept of technology assessment was created to evaluate the effects of the introduction or development of new technologies, especially focusing on the negative impacts. This concept responded to technologies emerging in the second half of the 20th century that were widely recognized as risky or dangerous, such as biotechnologies, nanotechnologies, and nuclear technologies [Goulet, 1994; Coates, 1998; Tran, Daim, 2008]. Technology assessment is intended to provide an early warning system and identify opportunities and risks for the use of a technology so that the legitimacy of its implementation and development can be verified. For many years, specialists have been tasked with making the public aware of the potential that new technologies have in order to convince people to implement the innovations [Halicka, 2017; Halicka, 2018].
From the literature review, it can be seen that initially the concept of Technology Assessment was used for political decision making. It was mainly used for strategic economic assessments of complex technologies, such as conventional and nuclear energy technologies and aeronautical technologies. Most of these technologies have been developed and implemented by government institutions. Over time, TA has been used for business decision making and the evaluation and selection of implemented technologies. A literature review shows that there are several varieties of TA (Table 1) [Carlsen et al., 2010; Tran, Daim, 2008; van den Ende et al., 1998].
Participatory Technology Assessment (PTA) is about increasing the participation and influence of the public in the decision-making process based on what is already known about a technology, rather than an-
ticipating the unexpected effects of future technologies [Goulet, 1994; Tavella, 2016]. Awareness TA (ATA), on the other hand, focuses on anticipating a technological change and its impact, with a particular focus on unplanned and unexpected consequences [Coates,1998; Arora et al., 2014]. Constructive TA (CTA) makes it possible to shape the course of a technological development in socially desirable directions [van den Ende et al., 1998; Schot, Rip, 1997; Versteeg el al., 2017]. Backcasting is about developing scenarios for the desired future and launching innovative processes based on these scenarios [Zimmermann et al., 2012]. Strategic TA (STA) supports specific entities or groups of entities in formulating their policies and strategies for specific technological developments [Daim et al., 2018; Grimaldi et al., 2015]. The first four types of TA are currently used for political decision making. The last type of technology assessment (STA) points to the emergence of a further stream of TA research in the business, industry, and non-governmental environments.
The strategic technology assessment was first carried out in the 1980s, but a more detailed version followed after the 1990s. This approach can be used for the economic assessment of alternative technologies, for the selection and purchase of strategic technologies, or strategic planning. It considers technological readiness, commercial potential, or innovative technology. In this case, the dominant role is played by a potential or current supplier or user interested in the commercialization or implementation of the most appropriate technologies. Therefore, the technology assessment can be made by organizations developing technologies (e.g. research institutes) or enterprises that want to select and implement the most appropriate technologies for their needs. In this trend of technology assessment, the importance of technology is examined from the point of view of the recipient, current and expected plans for its implementation, and the application by the recipient and other entities. This trend in technology assessment is an unsavory element of technology selection.
Literature Review
Technology selection is a dynamically developing area, which is reflected in a growing number of publications. Over the last 40 years, 1,753 publications have been indexed in the Scopus database with the keyword "technology selection". The number of publications between 1979 and 2018 is shown in Figure 1. According to the figure, initially — during the first twenty-five years — there was no significant interest in this issue. Until 2003, no more than 40 articles in this field were published annually. Only since 2004, has interest in the selection of technologies started to significantly increase, which is reflected in the number of publications in the Scopus database.
Figure 1. Number of Publications in the Scopus Database in 1979-2018
140 120 100 80 60 40 20
_
c-^cocococococococococo o\o\o\o\o\o\o\o\o\o\
(Nm^m^-ot^co
ÔÔÔÔÔÔÔÔÔÔOOOOOOOOO (N(N(N(N(N(N(N(N(N(N(N(N(N(N(N(N(N(N(N
o^fNm^m^-ot^co^o 0000000000
Source: own calculations based on the search results in the Scopus database for the term "technology selection" in the title, abstract, or the keywords of an article.
0
The largest number of publications on technology selection was announced in journals such as SAE Technical Papers (31 articles), A Journal Of Cleaner Production (31 articles), International Journal of Production Research (18 articles) and the International Journal of Advanced Manufacturing Technology (16 articles).
For the selection of technologies, both qualitative and quantitative methods can be used. The first group is aimed at identifying features that may potentially impact the effect of implementation and commercialization. The second group of methods should be used to identify the relevant characteristics that explain the reasons for the differences between technologies. Practical methods are usually a combination of quali-
tative and quantitative approaches. Research conducted around the world suggests that it is impossible to choose one method, which is best suited for technology analysis. Consequently, there is a noticeable trend in the use of several methods in each procedure.
The process of evaluating and selecting technologies is difficult. The reasons for this arise from the uncertainty surrounding the production of technology, including the ambiguity of the assessments (judgements) of the experts involved in the ranking research, the interdependencies between technologies, and the multidimensional nature of technologies. Considering the specific features listed above, multiattribute decision-making methods are used to solve the problem of the selection and ranking of technolo-
Table 1. Types of TA
Types of TA Publications Application Recipient's Profile Selected Criteria for Technology Assessment
Participatory Technology Assessment (PTA) [Goulet, 1994; Tavella, 2016] to make political decisions policymakers • the economic value of a technology • opportunities to obtain advantages based on differentiation • opportunities to influence technological progress through government intervention • appropriateness of government intervention • low potential for misappropriation • significant social benefits
Sustainability Assessment of Technologies (SAT) [Ren et al., 2017]
Awareness Assessment of Technologies (ATA) [Coates,1998; Arora et al., 2014]
Constructive Assessment of Technologies (CTA) [van den Ende et al., 1998; Schot, Rip, 1997; Versteeg el al., 2017]
Backcasting [Zimmermann et al., 2012]
Strategic Assessment of Technologies (STA) [Daim et al., 2018; Grimaldi et al., 2015] to make business decisions decisionmakers • validity from the point of view of the recipient • current implementation/application plans • expected implementation/application plans • time of market introduction • number of suppliers/points of sale
Source: author's study based on [Goulet, 1994; Tavella 2016; Ren et al., 2017; Coates,1998; Arora et al., 2014; van den Ende et al., 1998; Schot, Rip, 1997; Versteeg el al., 2017; Zimmermann et al., 2012; Daim et al., 2018; Grimaldi et al., 2015].
Table 2. Number of Publications in the Scopus Database between 1999-2019
Year Number of publications
1999 1
2009 1
2011 1
2012 4
2013 3
2014 1
2015 2
2016 5
2017 5
2018 6
2019 4
Source: own calculations based on the search results in the Scopus database for keywords "technology selection" and "TOPSIS", or "technology assessment" AND "TOPSIS" in the title, abstract, or keywords of an article.
gies [Winebrake, Creswick, 2003]. MADM methods define procedures for processing the information on the value (assessment) of options against criteria to prioritize solutions and select the best one. Using the approach described above, a wide range of methods was distinguished, including [Zavadskas et al., 2016; Mardania et al., 2018; Vavrek, Adamisin, Kotulic, 2017; Tamosiunas, 2018; Roszkowska, Kacprzak, 2016; Chodakowska, Nazarko, 2017]: the SAW (Simple Additive Weighting) method, ELECTRE (Elimination and Choice Expressing the Reality) and PROMETHEE (Preference Ranking Organization Method for Enrichment Evaluation) ranking methods, as well as methods based on the degree of distance
Table 3. Breakdown of Publications by Subject Matter of the Identified Articles
Subject Area Number of Publications
Engineering 19
Computer Science 8
Environmental Science 7
Business 5
Energy 3
Medicine 4
Social Sciences 2
Decision Sciences 2
Materials Science 2
Agricultural and Biological Sciences
Biochemistry
Chemistry
Mathematics
Physics and Astronomy
Source: own study based on records from the Scopus database.
from the ideal/anti-ideal VIKOR (VIsekriterijumska optimizacija i KOmpromisno Resenje — Multi-criteria Optimization and Compromise Solution). The most important of these are Multicriteria Optimisation and Compromise Solution, Technique for Order Preference by Similarity to Ideal Solution, Analytic Hierarchy Process, ANP (Analytic Network Process), and MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique). The literature review shows that the AHP and TOPSIS methods are most frequently used to select technologies. A characteristic feature of the AHP method is that it compares the adopted criteria with each other, which results in a comparison matrix. The next step in the AHP method is to determine global and local preferences based on a comparison matrix and to calculate the compliance factor. The final step is to create a final ranking of the accepted alternatives. This is possible by calculating the usefulness function of the variants. The TOPSIS method, on the other hand, is a method of similarity to an ideal solution, which is classified as a distance method. The variants are evaluated by determining their distance from the ideal and anti-master. The determination of the preferential sequence requires the consideration of the weights of the criteria and the standardization of the assessment of the alternatives in the light of the criteria. The best solution is the one closest and the one furthest from the ideal. This allows for determining the value of a synthetic meter, which indicates the position of particular variants in the ranking. AHP methods are not usually used in situations with a large number of criteria. For example, for 24 criteria, the matrix has 24 columns and 24 rows. It is usually used when there are less than 10 criteria. Moreover, in the AHP method, weights for particular criteria are often determined subjectively, based on expert opinions. Moreover, problems frequently result from interdependencies between alternatives and criteria. This may lead to inconsistencies between the decision and ranking criteria and the reversal of the ranking [Nermed, 2015; Velasquez, Hester, 2013; Anand, Vinodh, 2018; Mobinizadeh et al., 2016; Oztaysi, 2014]. Therefore, this study uses the TOPSIS method to select road technologies.
Initially, a detailed literature review was carried out and a bibliographic analysis of publications on technology selection using the TOPSIS method was performed. In the Scopus database for the period 1999-2019, 33 records are indexed with the keywords "technology selection" and "TOPSIS" or "technology assessment" AND "TOPSIS". The number of publications is presented in Table 2. The first articles in this field were published in 1999.
The identified publications were analyzed in terms of subject areas (Table 3). Each article could be assigned to several areas. More than half of the identified publications concerned engineering issues. Issues attributed to the area of Computer Science were addressed
Table 4. TOPSIS Method in Technology Selection
Authors (year) Type of Technology
[Habbal et al., 2019] radio access technologies
[Gladysz et al., 2017; Wan et al., 2016] radio frequency identification (RFID)
[Zhang et al., 2019] energy storage technology
[Restrepo-Garces et al., 2017; Hirushie et al., 2017] renewable energy sources
[Karatas et al., 2018] energy technology
[Streimikiene, 2013a,b; Streimikiene et al., 2013; Streimikiene, Balezentiene, 2012] electric vehicles
[Zheng et al., 2012] green buildings
[Peng et al., 2019] restoration technology in engine remanufacturing practice
[Aloini et al., 2018] advanced underwater system
[Buyukozkan, Guler, 2017] smart glass (SG)
[Ansari et al., 2016; Puthanpura et al., 2015] automotive technology
[Elahi et al., 2011] ABS sensor technology
[Govind et al., 2018] treatment and disposal technology
[Ren, 2018] ballast water treatment
[Vivekh et al., 2017] desalination technology
[Kalbar et al., 2012; Fu et al., 2012] wastewater treatment technology
[Jin, 2018; Mobinizadeh et al., 2016; Gajdos et al., 2015; Lu et al., 2016] health technology
[Lee, James Chou, 2016] emerging three-dimensional integrated circuit (3DIC)
[Tavana et al., 2013] advanced-technology projects at NASA
[Oztaysi, 2014] information technology
[Towhidi et al., 2009] iron-making technology
[Parkan, Wu, 1999] robots to perform repetitious, difficult, and hazardous tasks with precision
Source: own study.
(3DIC), or iron making as well as advanced underwater systems.
This article looks into the use of the TOPSIS method to rank the following five road-pavement technologies [Nazarko et al., 2015; Nazarko, 2017; Kikolski, Chien-Ho Ko, 2018]: road pavement with rubber-asphalt binder (T1), pavement with porous asphalt mixture (T2), Perpetual Pavement (T3), the traditional cement concrete (T4), and pavement with elastomeric binders (T5). Literature studies and exploratory research conducted gave rise to the following research questions: (1) How does one apply the TOPSIS method to the assessment of road pavement technology? (2) What are the criteria for assessing road pavement technology? (3) How has technology been assessed against various criteria?
Table 5. Scheme for the Operationalization of the Assessment and Selection of Road Pavement Technologies
Research Task Contractor Method Result
1. Assessment of Technology Maturity and Performance The author, experts Literature review, Technology Readiness Levels, life cycle analysis Life cycle phases of technologies, levels of technological maturity
2. Identification of Technology Assessment Criteria The author Literature review The criteria catalog
3. Technology Assessment Experts Surveys Completed technology assessment questionnaires
4. Technology Selection The author TOPSIS Ranking
Source: compiled by the author.
in eight articles, and seven articles were dedicated to the area of Environmental Science. Other articles dealt with Business, Energy, Material Sciences, and Social Sciences.
The review of the publications shows that the TOPSIS method was used most frequently to select energy technologies, such as energy storage or renewable sources as well as health technology (Table 4). This method was also used to rank environmental technologies (i.e., treatment and disposal, ballast water treatment, desalination, wastewater treatment, healthcare waste treatment technologies) and automotive industry technologies (i.e., the restoration in engine remanufacturing practice, ABS sensors). It was also used for such technologies as smart glass (SG), emerging three-dimensional integrated circuit
Research Methods
The process of road pavement technology selection was carried out in four successive stages. The operational diagram of road pavement technology selection is presented in Table 5.
Completing the first research task, the author and key field experts assessed the level of technological maturity of the prioritized road surface technologies in Poland. The model of Technology Readiness Levels (TRL) was applied for this purpose. According to the literature, this technology has a low level of technological readiness. In the case of the three levels of technological readiness, the basic principles of the analyzed technology were observed and described, the concept of the technology and/or its application was defined, and the key functions and/or the concept of the technology were confirmed analytically and experimentally. Technologies with a medium level of technological preparedness have TRL 4, 5, and 6. Technologies with a medium level of preparedness have already been tested in a near-real environment. Technologies with a high level of technological readiness have TRL 7, 8, and 9. Technologies with a low to medium level of technological readiness include fundamental research consisting of the acquisition
Table 6. Catalogue of Technology Assessment Criteria
Acronym Name of the Criterion
TRL Technology Readiness Levels
S S-life-cycle analysis
Innovation
11 What is the level of technological innovation?
12 Is the technology original according to the current state of knowledge?
13 Is there an improvement in the technology compared to existing alternatives?
Competitiveness
C1 Is the market position of the technology threatened by existing solutions?
C2 How will the dissemination of the technology affect the existing alternative solutions?
C3 Are the new opportunities offered by the technology compared to the alternatives relevant for road users?
C4 Is the improvement in the comfort of use compared to the alternatives to the technology relevant for road users?
C5 How many similar alternatives to technologies are available on the Polish market?
C6 What is the popularity of the alternatives to the technology?
C7 Are there entry barriers for potential competitors?
Usability
U1 Does the technology have measurable value for users?
U2 Will potential users gain additional benefits from the use of the technology that are not available when alternatives are used?
U3 Does the technology or the product based on it offer higher user-friendliness and ease of use than the available alternatives?
U4 Is the technology or product based on it compliant with the formal requirements applicable in Poland and the European Union?
U5 Can the demand for a technology or a product based on it be related to transitional fashion?
U6 Do recent changes in the environment make the technology or a product based on it more attractive to users (for example, due to new legislation, consumer trends, or technological standards)?
U7 At what point in time may the technology or product based on it become obsolete?
U8 Will the technology solve technical problems that are perceived as important by potential customers?
U9 Are the technical benefits offered by the technology important to potential customers?
U10 Are potential customers sensitive to the possible technical problems related to the use of the technology?
Source: own study.
of new knowledge of the underlying principles and observable facts, without a direct practical or industrial application focus. This is aimed at acquiring new knowledge and skills to develop new products, processes, and services or to bring a significant improvement to existing products, processes, and services. The phases of the life cycle of individual technologies are then determined. The following life phases of a technology are listed in the literature: the birth phase, which is characterized by high uncertainty, research intensity, and the reduction of investment; the development phase, which is characterized by average uncertainty, an emphasis upon applications, and high investment; the maturity phase, which is characterized by low uncertainty, cost reduction, and the reduction of investment as well as a decline in technology assessment where the technology is outdated and replaced by a new technology with a higher competitive value.
During the second task, three groups of technology assessment criteria were selected on the basis of a literature review [Ejdys et al, 2016, Ejdys, 2015]: (1) innovation, (2) competitiveness, and (3) usability. The criteria were developed in the form of questions. The author's catalogue of criteria consisted of
Table 7. Decision Matrix
TRL S I1 I2 I3 K1 K2 K3 K4 K5 K6
T1 8 2 4 4 4 1 3 4 4 2 1
T2 8 2 4 1 4 3 3 4 4 3 1
T3 7 1 5 3 4 5 4 5 5 5 5
T4 9 3 1 1 1 1 3 3 1 2 1
T5 9 2 4 3 3 3 2 4 3 3 4
K7 U1 U2 U3 U4 U5 U6 U7 U8 U9 U10
T1 3 5 5 3 4 4 4 5 4 4 3
T2 4 3 3 4 4 4 4 5 4 4 1
T3 3 5 5 4 5 4 4 5 5 5 4
T4 4 3 2 1 5 2 3 5 3 3 2
T5 5 4 1 1 5 5 1 5 4 4 4
Source: compiled by the author.
Step 1. Initially, there was a set of criteria consisting of 24 elements:
22 questions. Three questions concerned innovation (I1 — I3), seven questions concerned technological competitiveness (C1 — C7), and ten questions concerned technological usability (U1 — U10). The list of criteria used for the assessment of road pavement technologies is given in Table 6. The selection of road pavement technologies has not been carried out so far. This is the first (pilot) study of this type in Poland. One important goal of the study is to develop an objective ranking of road pavement technologies. It was imperative that the position of a given technology in the ranking should be determined only by the properties of a given technology in the context of a given criterion. Therefore, decision-makers were not consulted regarding the importance of the assessment criteria for these technologies. Also, a conscious decision was made not to assign weights to the criteria based on the opinion of the experts or decision-makers. Rather, the weights of the criteria were developed using the entropy method. The entropy method makes it possible to estimate the importance of analyzed criteria describing the considered solution options based on each of their value discrepancies [Lotfi, Fallahnejad, 2010; Kacprzak, 2017].
Next, experts evaluated the assessment of the analyzed technology using the 5-point Likert scale, where 1 was the lowest score and 5 was the highest score. Each expert assessed one technology. The experts were selected purposively, considering their knowledge and experience in the field of road surface technology in Poland. Employees of the Warsaw University of Technology participating in the Team of Materials and Road Surfaces Technology were invited to be the experts. Then, during the fourth task, using the TOPSIS method, a ranking of road pavement technologies was developed.
Research Results
The TOPSIS technology ranking [Hwang, Yoon, 1981] was achieved in seven consecutive steps, as shown below.
{C, j = 1, ... n}
(1)
The first criterion was the TRL, followed by life-cycle phases of technologies, three further criteria for technology innovation, seven more for competitiveness, and ten more for usability. The TRL could range from 1 to 9, life-cycle phases of technologies — from 1 to 4, and the remaining criteria — from 1 to 5.
Step 2. Then, based on the assessment of technology by experts in terms of the subsequent criteria, a decision matrix was developed (Table 7):
X = (j (2)
where x..eR
X =
i = 1, ... m; j = 1, ... n
(3)
The analysis of Table 7 shows that T1 technology had a TRL of 8 and the second life-cycle phases of the technology. The T1 technology was assessed by an expert as regards the criterion I1 at the level 4, as well as the criterion K1 - 1 and the criterion U1 - 5 (on a scale from 1 to 5). The T4 technology, on the other hand, was assessed in terms of criteria I1 and K1 at level 1, while also evaluated in terms of the criterion U1 at level 3.
Step 3. A normalized (vector-based) decision matrix (Table 8) was then developed:
(4)
(5)
(6)
R = (r), r
R =
where
J
ymx 2
, = 1 j
Step 4. The next step was to determine the criterion weight vector (Table 9). For this purpose, the entropy method was used [Kacprzak, 2017; Rudnik, Kacprzak, 2017]:
E = (ev e2' ■■■
where E — an entropy vector, and
1
e, = ■
j
^ m
lnm i -
z.. ln z. .,
1 ij ij
and
z lnz = 0, where z = 0,
j j j
with a vector of criteria weights:
w = (w,, W., ■■■, w ),
1 1 2 n
(7)
(8)
9) (10)
x
x
x , ... x
m1 mn
Table 8. Normalized Decision Matrix
TRL S I1 I2 I3 K1 K2 K3 K4 K5 K6
T1 0.435 0.426 0.465 0.667 0.525 0.149 0.438 0.442 0.489 0.280 0.151
T2 0.435 0.426 0.465 0.167 0.525 0.447 0.438 0.442 0.489 0.420 0.151
T3 0.380 0.213 0.581 0.500 0.525 0.745 0.583 0.552 0.611 0.700 0.754
T4 0.489 0.640 0.116 0.167 0.131 0.149 0.438 0.331 0.122 0.280 0.151
T5 0.489 0.426 0.465 0.500 0.394 0.447 0.292 0.442 0.367 0.420 0.603
K7 U1 U2 U3 U4 U5 U6 U7 U8 U9 U10
T1 0.346 0.546 0.625 0.457 0.387 0.456 0.525 0.447 0.442 0.442 0.442
T2 0.462 0.327 0.375 0.610 0.387 0.456 0.525 0.447 0.442 0.442 0.147
T3 0.346 0.546 0.625 0.610 0.483 0.456 0.525 0.447 0.552 0.552 0.590
T4 0.462 0.327 0.250 0.152 0.483 0.228 0.394 0.447 0.331 0.331 0.295
T5 0.577 0.436 0.125 0.152 0.483 0.570 0.131 0.447 0.442 0.442 0.590
Source: compiled by the author.
w. <
1
[0, 1], 1
j = I",
1,
(11)
where w, — the criterion weight. If all the criteria were equally valid, the weights were calculated according to the formula:
d.
w. = —
1 I
d = 1 i
: 1 d.
(12)
(13)
Aiming to determine entropy, the decision matrix should be normalized:
Z
(z),
Z =
where
x..
1 x..
1 i1
(14)
(15)
(16)
All the weight factors are presented in Table 9.
Table 9 shows the most important criteria: K6 (w=0.164), K1 (w=0.109), and U3 (w=0.097). The least important criteria were U7 (w=0.000), TRL (w=0.003), and U4 (w=0.004).
Weight factors were determined and the weighted normalized decision matrix (Table 10) was developed:
V = (J
where
v.. = r.. w.
'1 '1 1
(17)
(18)
Step 5. The next step involved the recognition of the positive-ideal solution A+ and the negative-ideal solution A [Kacprzak, 2019].
A+ = [v+, v2 +, ..., vn]= [max v max v ... max v.3] (19)
A = [v ", v27 ..., vn]= [min v min .v.2 ... min vi3] (20)
After selecting the distance measure, the separation measures s + and s- of each alternative were calculated from the intuitionistic fuzzy positive-ideal and the negative-ideal solutions. This paper used the normalized Euclidean distance:
. =J I n = (v+ -
1 = J Z n= (v- - v^
(21) (22)
Step 6. Then, the relative closeness coefficient is calculated. The relative closeness coefficient of an alternative A, with respect to the positive-ideal solution A+ is defined as follows:
C =
s.
s/ + s-'
(23)
where 0 ^ C.s 1.
e
z
z
11
z
z
m1
z
Table 9. Weights of the Evaluation Criteria
TRL S I1 I2 I3 K1 K2 K3 K4 K5 K6
e 0.997 0.967 0.944 0.916 0.949 0.894 0.986 0.992 0.940 0.961 0.840
d 0.003 0.033 0.056 0.084 0.051 0.106 0.014 0.008 0.060 0.039 0.160
w 0.003 0.033 0.058 0.087 0.053 0.109 0.014 0.008 0.061 0.040 0.164
K7 U1 U2 U3 U4 U5 U6 U7 U8 U9 U10
e 0.988 0.984 0.916 0.906 0.996 0.977 0.949 1.000 0.992 0.992 0.940
d 0.012 0.016 0.084 0.094 0.004 0.023 0.051 0.000 0.008 0.008 0.060
w 0.012 0.016 0.087 0.097 0.004 0.024 0.053 0.000 0.008 0.008 0.062
Source: compiled by the author.
Table 10. Weighted Normalized Decision Matrix
TRL S I1 I2 I3 K1 K2 K3 K4 K5 K6
T1 0.001 0.014 0.027 0.058 0.028 0.016 0.006 0.004 0.030 0.011 0.001
T2 0.001 0.014 0.027 0.014 0.028 0.049 0.006 0.004 0.030 0.017 0.001
T3 0.001 0.007 0.034 0.043 0.028 0.081 0.008 0.004 0.037 0.028 0.001
T4 0.001 0.021 0.007 0.014 0.007 0.016 0.006 0.003 0.007 0.011 0.001
T5 0.001 0.014 0.027 0.043 0.021 0.049 0.004 0.004 0.022 0.017 0.001
K7 U1 U2 U3 U4 U5 U6 U7 U8 U9 U10
T1 0.025 0.004 0.009 0.054 0.044 0.001 0.011 0.028 0.000 0.004 0.004
T2 0.025 0.006 0.005 0.032 0.059 0.001 0.011 0.028 0.000 0.004 0.004
T3 0.124 0.004 0.009 0.054 0.059 0.002 0.011 0.028 0.000 0.004 0.004
T4 0.025 0.006 0.005 0.022 0.015 0.002 0.005 0.021 0.000 0.003 0.003
T5 0.099 0.007 0.007 0.011 0.015 0.002 0.013 0.007 0.000 0.004 0.004
Source: compiled by the author.
C measures the effectiveness of each alternative. The
i
best alternative and the order of the alternatives are obtained according to this measure.
Step 7. Once the relative closeness coefficient of each alternative is determined, alternatives are ranked in the descending order of C. [BoranGenf et al., 2009; Yue, 2014].
As a result (Table 11), the Perpetual Pavement (T3) was found to be the most desirable product among these alternatives, overtaking its nearest competitor, pavement with elastomeric binders (T5). Rubber-asphalt binder (T1) ranked third, followed by the pavement with porous asphalt mixture (T2), leaving the traditional cement concrete (T4) last.
Conclusion
The paper presents a proposal to apply the TOPSIS method to the assessment and selection of road pavement technologies, such as road pavement with rubber-asphalt binder (T1), pavement with a porous
Table 11. Relative Closeness and the Preferential
Ranking of Alternative Options
Road Pavement Technology S+ } S. } C. Rank
T1 0.121653105 0.082644528 0.40453 3
T2 0.119536825 0.073261956 0.37999 4
T3 0.020692762 0.149284031 0.87826 1
T4 0.147440469 0.024541157 0.14270 5
T5 0.081536767 0.095672852 0.53989 2
Source: compiled by the author.
asphalt mixture (T2), the Perpetual Pavement (T3), the traditional cement concrete (T4), and pavement with elastomeric binders (T5). Initially, based on the literature, the maturity and efficiency of the five road technologies were evaluated. Then, technology selection criteria were identified for the assessment of innovation, competitiveness, and usefulness. Experts evaluated the technology considering the level of maturity and efficiency of the technology and the 22 criteria identified on the basis of the literature. The TOPSIS method was followed by a ranking of the best road pavement technologies. T3 — the Perpetual Pavement was the best of the assessed technologies. T4 — the traditional cement concrete technology was ranked last.
The conducted research found answers to the following research questions: (1) How does one apply the TOPSIS method to the assessment of road pavement technology? (2) What are the criteria for assessing road pavement technology? (3) How has the technology been assessed against various criteria?
It can also be argued that the present method of decision making can also be used effectively in a more complex analysis.
In future studies, when constructing the ranking, the opinions of decision-makers regarding the substance of the criteria will be considered. It is also planned to extend the study to other European countries and compare road pavement technology rankings in different countries. It is also planned to expand the catalogue of criteria and develop rankings using other methods.
This research was conducted within the scope of the Project S/WZ/1/2017 and financed by the Ministry of Science and Higher Education.
References
Aloini D., Dulmin R., Mininno V., Pellegrini L., Farina G. (2018) Technology assessment with IF-TOPSIS: An application in the advanced underwater system sector. Technological Forecasting and Social Change, vol. 131, pp. 38-48. DOI: 10.1016/j. techfore.2017.07.010.
Anand M.B., Vinodh S. (2018) Application of fuzzy AHP - TOPSIS for ranking additive manufacturing processes for microfabrication. Rapid Prototyping Journal, vol. 24, no 2, pp. 424-435. DOI: 10.1108/RPJ-10-2016-0160.
Ansari R., Soltanzadeh J., Tavassoli A. (2016) Technology selection between technology management and decision making: A case study from the Iranian automotive industry. International Journal of Automotive Technology and Management, vol. 16, no 4, pp. 365-388. DOI:10.1504/IJATM.2016.081618.
Arora S.K., Foley R.W., Youtie J., Shapira P., Wiek A. (2014) Drivers of technology adoption - the case of nanomaterials in building construction. Technological Forecasting and Social Change, vol. 87, pp. 232-244. DOI: 10.1016/j.techfore.2013.12.017.
Boran F.E., Genç S., Kurt M., Akay D. (2009) A multi-criteria intuitionistic fuzzy group decision making for supplier selection with TOPSIS method. Expert Systems with Applications, vol. 36, no 8, pp. 11363-11368.
Buyukôzkan G., Guler M. (2017) A hesitant fuzzy based TOPSIS approach for smart glass evaluation. Proceedings of: EUSFLAT-2017 - The 10th Conference of the European Society for Fuzzy Logic and Technology, September 11-15, 2017, Warsaw, Poland (eds. J. Kacprzyk, E. Szmidt, S. Zadrozny, K. Atanassov, M. Krawczak), Heidelberg, Dordrecht, London, New York: Springer, pp. 330-341. DOI: 10.1007/978-3-319-66830-7_30.
Carlsen H., Dreborg K.H., Godman-Hansson S.O., Johansson L., Wikman-Svahn P. (2010) Assessing socially disruptive technological change. Technology in Society, vol. 32, pp. 209-218. Available at: https://doi.org/10.1016/j.techsoc.2010.07.002, accessed 14.03.2019.
Chodakowska E., Nazarko J. (2017) Environmental DEA method for assessing productivity of European countries. Technological and Economic Development of Economy, vol. 23, no 4, pp. 589-607. DOI: 10.3846/20294913.2016.1272069.
Coates J.F. (1998) Technology assessment as guidance to governmental management of new technologies in developing countries. Technological Forecasting and Social Change, vol. 58, no 1-2, pp. 35-46. DOI: 10.1016/S0040-1625(97)00087-5.
Daim T.U., Yoon B.S., Lindenberg J., Grizzi R., Estep J., Oliver T. (2018) Strategic roadmapping of robotics technologies for the power industry: A multicriteria technology assessment. Technological Forecasting and Social Change, vol. 131, pp. 49-66. DOI: 10.1016/j.techfore.2017.06.006.
Ejdys J. (2015) Innovativeness of residential care services in Poland in the context of strategic orientation. Procedia - Social and Behavioral Sciences, vol. 213, pp. 746-752. DOI: 10.1016/j.sbspro.2015.11.461.
Ejdys J., Matuszak-Flejszman A., Szymanski M., Ustinovicius L., Shevchenko G., Lulewicz-Sas A. (2016) Crucial factors for Improving the ISO14001 Environmental Management System. Journal of Business Economics and Management, vol. 17, no 1, pp. 52-73. DOI: 10.3846/16111699.2015.1065905.
Elahi M., Alvandi M., Valehzagharad H.K., Memarzade M. (2011) Selecting the best ABS sensor technology using fuzzy MADM. Scientific Research and Essays, vol. 6, no 31, pp. 6487-6498. DOI:10.5897/SRE11.1079.
Fu J., Xie L., Qu M., Liang G., Ma X., Tang J., Zhang R., Bai Y. (2012) The application of entropy weight TOPSIS method to the optimization of wastewater treatment technology in livestock and poultry slaughtered plant. Shenyang Jianzhu Daxue Xuebao (Ziran Kexue Ban)/Journal of Shenyang Jianzhu University (Natural Science), vol. 28, no. 5, pp. 909-914.
Gajdos O., Jurickova I., Otawova R. (2015) Health technology assessment models utilized in the chronic care management. Proceedings of the Third International Conference, IWBBIO 2015, Granada, Spain, April 15-17,2015 (eds. F. Ortuno, I. Rojas), Heidelberg, Dordrecht, London, New York: Springer, pp. 54-65.
Gladysz B., Nalepa K., Santarek K. (2017) Justification of RFID implementation. A case study of white goods manufacturer. Management and Production Engineering Review, vol. 8. no 4, pp. 91-104. DOI: 10.1515/mper-2017-0040.
Govind Kharat M., Murthy S., Jaisingh Kamble S., Raut R.D., Kamble S.S. (2018) Fuzzy multi-criteria decision analysis for environmentally conscious solid waste treatment and disposal technology selection. Technology in Society, vol. 57, pp. 20-29. DOI:10.1016/j.techsoc.2018.12.005.
Goulet D. (1994) Participatory Technology-Assessment - Institutions and Methods. Technological Forecasting and Social Change, vol. 45. no 1, pp. 47-61. DOI: 10.1016/0040-1625(94)90062-0.
Grimaldi M., Cricelli L., Di Giovanni M., Rogo F. (2015) The patent portfolio value analysis: A new framework to leverage patent information for strategic technology planning. Technological Forecasting and Social Change, vol. 94, pp. 286-302. DOI: 10.1016/j.techfore.2014.10.013.
Habbal A., Goudar S.I., Hassan S. (2019) A context-aware radio access technology selection mechanism in 5G mobile network for smart city applications. Journal of Network and Computer Applications, vol. 135, pp. 97-107. DOI: 10.1016/j. jnca.2019.02.019.
Habbal A., Goudar S.I., Hassan S. (2017) Context-aware radio access technology selection in 5G ultra dense networks. IEEE Access, vol. 5, pp. 6636-6648. DOI:10.1109/ACCESS.2017.2689725.
Halicka K. (2018) The reference methodology of prospective analysis of technology in production engineering. 8th International Conference on Engineering, Project, and Product Management (EPPM2017) Proceedings (ed. S. §ahin), Heidelberg, Dordrecht, London, New York: Springer, pp. 99-107. DOI:10.1007/978-3-319-74123-9_11.
Halicka K. (2017) Main Concepts of Technology Analysis in the Light of the Literature on the Subject. Procedia Engineering, vol. 182, pp. 291-298.
Hirushie K., Kasun H., Rehan S. (2017) Renewable energy technology selection for community energy systems: A case study for British Columbia. Paper presented at the 2017 CSCE Annual General Conference, Vancouver, BC, Canada. Available at: https://www.researchgate.net/publication/326211412_Renewable_energy_technology_selection_for_community_energy_ systems_A_case_study_for_British_Columbia, accessed 26.04.2019.
Hwang C.L., Yoon K. (1981) Multiple Attribute Decision Making: Methods and Applications, Berlin: Springer-Verlag.
Jirí M. (2018) The robustness of TOPSIS results using sensitivity analysis based on weight tuning. IFMBE Proceedings, vol. 68, no 2, pp. 83-86. DOI:10.1007/978-981-10-9038-7_15.
Kacprzak D. (2019) A doubly extended TOPSIS method for group decision making based on ordered fuzzy numbers. Expert Systems with Applications, vol. 116, pp. 243-254. DOI: 10.1016/j.eswa.2018.09.023.
Kacprzak D. (2017) Objective Weights Based on Ordered Fuzzy Numbers for Fuzzy Multiple Criteria Decision Making Methods. Entropy, vol. 19, no 373. DOI:10.3390/e19070373. Available at: https://pdfs.semanticscholar. org/9ff9/552d78357f946c5fd35b4822d5c6be693cac.pdf, accessed 12.04.2019.
Kalbar P.P., Karmakar S., Asolekar S.R. (2012) Selection of an appropriate wastewater treatment technology: A scenario-based multiple-attribute decision-making approach. Journal of Environmental Management, vol. 113, pp. 158-169. DOI: 10.1016/j. jenvman.2012.08.025.
Karatas M., Sulukan E., Karacan I. (2018) Assessment of Turkey's energy management performance via a hybrid multi-criteria decision-making methodology. Energy, vol. 153, pp. 890-912. DOI: 10.1016/j.energy.2018.04.051.
Kikolski M., Chien-Ho Ko (2018) Facility layout design - review of current research directions. Engineering Management in Production and Services, vol. 10, no 3, pp. 70-79. DOI: 10.2478/emj-2018-0018.
Lee Y., James Chou C. (2016) Technology evaluation and selection of 3DIC integration using a three-stage fuzzy MCDM. Sustainability (Switzerland), vol. 8, no 2, pp. 1-15. DOI: 10.3390/su8020114. Available at: https://ideas.repec.org/aZgam/ jsusta/v8y2016i2p114-d62915.html, accessed 23.04.2019.
Lotfi F.H., Fallahnejad R. (2010) Imprecise Shannon's Entropy and Multi Attribute Decision Making. Entropy, vol. 12, pp. 53-62.
Lu C., You J., Liu H., Li P. (2016) Health-care waste treatment technology selection using the interval 2-tuple induced TOPSIS method. International Journal of Environmental Research and Public Health, vol. 13, no 6. DOI: 10.3390/ijerph13060562.
Mobinizadeh M., Raeissi P., Nasiripour A.A., Olyaeemanesh A., Tabibi S.J. (2016) A model for priority setting of health technology assessment: The experience of AHP-TOPSIS combination approach. DARU, Journal of Pharmaceutical Sciences, vol. 24, pp. 1. DOI:10.1186/s40199-016-0148-7. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4827190/, accessed 11.05.2019.
Mardania A., Jusoha A., Halicka K., Ejdys J., Magruk A., Ungku Norulkamar U.A. (2018) Determining the utility in management by using multi-criteria decision support tools: A review. Economic Research / Ekonomska Istrazivanja, vol. 31, pp. 1666-1716. Available at: https://doi.org/10.1080/1331677X.2018.1488600, accessed 18.06.2019.
Nazarko J., Radziszewski P., Dçbkowska K., Ejdys J., Gudanowska A., Halicka K., Kilon J., Kononiuk A., Kowalski K., Król J., Nazarko t., Sarnowski M., Vilutiene T. (2015) Foresight Study of Road Pavement Technologies. Procedia Engineering, vol. 122, pp. 129-136.
Nazarko t. (2017) Future-Oriented Technology Assessment. Procedia Engineering, vol. 182, pp. 504-509. DOI:10.1016/j. proeng.2017.03.144.
Nermed K. (2015) Wielokryterialna metoda wektora preferencji jako narzçdzie wspomagaj^ce proces decyzyjny [Multi-Criteria Preference Vector Method (PVM) as a tool supporting the decision making process]. Przeglqd Statystyczny, vol. 62, no 1, pp. 93-115 (in Polish). Available at: http://cejsh.icm.edu.pl/cejsh/element/bwmeta1.element.desklight-0f4c6200-68b5-4c3f-b307-ad24fcb0eef4, accessed 18.04.2019.
Nouri F.A., Esbouei S.K., Antucheviciene J. (2015) A hybrid MCDM approach based on fuzzy ANP and fuzzy TOPSIS for technology selection. Informatica (Netherlands), vol. 26, no 3, pp. 369-388. DOI: 10.15388/Informatica.2015.53.
Oztaysi B. (2014) A decision model for information technology selection using AHP integrated TOPSIS-grey: The case of content management systems. Knowledge-Based Systems, vol. 70, pp. 44-54. DOI: 10.1016/j.knosys.2014.02.010.
Parkan C., Wu M. (1999) Decision-making and performance measurement models with applications to robot selection. Computers and Industrial Engineering, vol. 36, no 3, pp. 503-523.
Peng S., Li T., Li M., Guo Y., Shi J., Tan G.Z., Zhang H. (2019) An integrated decision model of restoring technologies selection for engine remanufacturing practice. Journal of Cleaner Production, vol. 206, pp. 598-610. DOI: 10.1016/j.jclepro.2018.09.176.
Puthanpura A.K., Khalifa R., Chan L. (2015) Assessing emerging automotive technologies for the future. 2015 Proceedings of PICMET '15: Management of the Technology Age, pp. 2113-2120. DOI:10.1109/PICMET.2015.7273223. Available at: https:// pdfs.semanticscholar.org/9fe1/3e67e3ba91d29f8c9c0a3787355bf9519ae6.pdf, accessed 22.04.2019.
Ren J. (2018) Technology selection for ballast water treatment by multi-stakeholders: A multi-attribute decision analysis approach based on the combined weights and extension theory. Chemosphere, vol. 191, pp. 747-760. DOI: 10.1016/j. chemosphere.2017.10.053.
Ren J.Z., Liang H.W., Chan F.T.S. (2017) Urban sewage sludge, sustainability, and transition for Eco-City: Multi-criteria sustainability assessment of technologies based on best-worst method. Technological Forecasting and Social Change, vol. 116, pp. 29-39. DOI: 10.1016/j.techfore.2016.10.070.
Restrepo-Garcés A.R., Manotas-Duque D.F., Lozano C.A. (2017) Método Hibrido Multicriterio-ROA, para la selección de fuentes de energía renovables: Caso de estudio centros comerciales [Multicriteria hybrid method - ROA, for the choice of generation of renewable sources: Case study in shopping centers]. Ingeniare, vol. 25, no 3, pp. 399-414. DOI:10.4067/S0718-33052017000300399 (in Spanish).
Roszkowska E., Kacprzak D. (2016) The fuzzy SAW and fuzzy TOPSIS procedures based on ordered fuzzy numbers. Information Sciences, vol. 369, pp. 564-584.
Rudnik K., Kacprzak D. (2017) Fuzzy TOPSIS method with ordered fuzzy numbers for flow control in a manufacturing system. Applied Soft Computing, vol. 52, pp. 1020-1041. DOI: 10.1016/j.asoc.2016.09.027.
Schot J., Rip A. (1997) The past and future of constructive technology assessment. Technological Forecasting and Social Change, vol. 54, no 2-3, pp. 251-268. DOI: 10.1016/S0040-1625(96)00180-1.
Streimikiene D. (2013a) Assessment of road transport technologies based on GHG emission reduction potential and costs. Transformations in Business and Economics, vol. 12, no 2, pp. 138-147.
Streimikiene D. (2013b) Assessment of energy technologies in electricity and transport sectors based on carbon intensity and costs. Technological and Economic Development of Economy, vol. 19, no 4, pp. 606-620. DOI: 10.3846/20294913.2013.837113.
Streimikiene D., Balezentiene L. (2012) Assessment of electricity generation technologies based on GHG emission reduction potential and costs. Transformations in Business and Economics, vol. 11, no 2A, pp. 333-343.
Streimikiene D., Balezentis T., Balezentiene L. (2013) Comparative assessment of road transport technologies. Renewable and Sustainable Energy Reviews, vol. 20, pp. 611-618. DOI: 10.1016/j.rser.2012.12.021.
Tamosiünas A. (2018) Managing selection of wind power generation technologies. Business: Theory and Practice, vol. 19, pp. 309-321. D0I:10.3846/btp.2018.31.
Tavana M., Khalili-Damghani K., Abtahi A. (2013) A hybrid fuzzy group decision support framework for advanced-technology prioritization at NASA. Expert Systems with Applications, vol. 40, no 2, pp. 480-491. DOI: 10.1016/j.eswa.2012.07.040.
Tavella E. (2016) How to make Participatory Technology Assessment in agriculture more "participatory": The case of genetically modified plants. Technological Forecasting and Social Change, vol. 103, pp. 119-126. DOI: 10.1016/j.techfore.2015.10.015.
Towhidi N., Tavakkoli-Moghaddam R., Peymandar M. (2009) Iron-making technology selection using a fuzzy hierarchical TOPSIS method. Proceedings of the 5th International Congress on the Science and Technology of Ironmaking (ICSTI) 2009, pp. 1039-1044.
Tran T.A., Daim T. (2008) A taxonomic review of methods and tools applied in technology assessment. Technological Forecasting and Social Change, vol. 75, pp. 1396-1405. DOI: 10.1016/j.techfore.2008.04.004.
van den Ende J., Mulder K., Knot M., Moors E., Vergragt P. (1998) Traditional and modern technology assessment: Toward a toolkit. Technological Forecasting and Social Change, vol. 58, no 1-2, pp. 5-21. DOI: 10.1016/S0040-1625(97)00052-8.
Vavrek R., Adamisin P., Kotulic R. (2017) Multi-Criteria Evaluation of Municipalities in Slovakia - Case Study in Selected Districts. Polish Journal of Management Studies, vol. 16, no 2, pp. 290-301. DOI: 10.17512/pjms.2017.16.2.25.
Velasquez M., Hester P.T. (2013) An Analysis of Multi-Criteria Decision Making Methods. International Journal of Operations Research, vol. 2, no 10, pp. 56-66.
Versteeg T., Baumann M.J., Weil M., Moniz A.B. (2017) Exploring emerging battery technology for grid-connected energy storage with Constructive Technology Assessment. Technological Forecasting and Social Change, vol. 115, pp. 99-110. DOI: 10.1016/j.techfore.2016.09.024.
Vivekh P., Sudhakar M., Srinivas M., Vishwanthkumar V. (2017) Desalination technology selection using multi-criteria evaluation: TOPSIS and PROMETHEE-2. International Journal of Low-Carbon Technologies, vol. 12, no 1, pp. 24-35.
Wan S.-P., Wang F., Dong J.-Y. (2016) A novel group decision making method with intuitionistic fuzzy preference relations for RFID technology selection. Applied Soft Computing Journal, vol. 38, pp. 405-422. DOI: 10.1016/j.asoc.2015.09.039.
Winebrake J.J., Creswick B.P. (2003) The future of hydrogen fueling systems for transportation: An application of perspective-based scenario analysis using the analytic hierarchy process. Technological Forecasting and Social Change, vol. 70, no 4, pp. 35-384. DOI: 10.1016/S0040-1625(01)00189-5.
Zavadskas E.K., Mardani A., Turskis Z., Jusoh A., Nor K.M.D. (2016) Development of TOPSIS method to solve complicated decision-making problems: An overview on developments from 2000 to 2015. International Journal of Information Technology and Decision Making, vol. 15, no 3, pp. 645-682.
Zhang C., Chen C., Streimikiene D., Balezentis T. (2019) Intuitionistic fuzzy MULTIMOORA approach for multi-criteria assessment of the energy storage technologies. Applied Soft Computing Journal, vol. 79, pp. 410-423. DOI: 10.1016/j. asoc.2019.04.008.
Zheng G., Jing Y., Huang H., Zhang X. (2011) Multihierarchical gray evaluation method to assess building energy conservation. Journal of Energy Engineering, vol. 137, no 2, pp. 88-98. DOI:10.1061/(ASCE)EY.1943-7897.0000041.
Zimmermann M., Darkow I.L., von der Gracht H.A. (2012) Integrating Delphi and participatory backcasting in pursuit of trustworthiness - The case of electric mobility in Germany. Technological Forecasting and Social Change, vol. 79, no 9, pp. 1605-1621. DOI: 10.1016/j.techfore.2012.05.016.
Yue Z. (2014) TOPSIS-based group decision-making methodology in intuitionistic fuzzy setting. Information Sciences, vol. 277, pp. 141-153.
