CC BY
3
COMPARATIVE ANALYSIS OF THE INDICATORS OF THE RESEARCH FACTOR OF LATIN AMERICAN UNIVERSITIES IN THE TOP 10 OF THE
SIR IBER 2020 RANKING
1HUGO ROLANDO SÁNCHEZ QUISPE, 2MICHAEL ESTAFANIA JÁTIVA BRITO,3SANTIAGO ISRAEL LOGROÑO NARANJO, 4NÉSTOR AUGUSTO ESTRADA BRITO
1Escuela Superior Politécnica de Chimborazo- Sede Orellana ORCID: 0000-0001-9785-9337 [email protected] 2Escuela Superior Politécnica de Chimborazo- Sede Orellana
[email protected] 3Escuela Superior Politécnica de Chimborazo- Sede Orellana
[email protected] 4Escuela Superior Politécnica de Chimborazo- Sede Morona [email protected]
Abstract: The SIR Iber shows the scientific activity of universities in Andorra, Spain, Portugal, and Latin American countries by positioning them based on measurement, research, innovation, and social impact factors (through a system of indicators). In this paper, the twelve (12) indicators of the research measurement factor of Latin American universities in the TOP 10 of the SIR Iber 2020 Ranking are analyzed and compared. Although the USP (Brazil) ranks first in Latin America, the University of Chile ranks first in the International Collaboration (IC), Normalized Impact (NI), Weighted Citation Normalized (WCN), High Quality Publications (Q1), Excellence (EXC), and Excellence with Leadership (EwL). A scatter diagram is presented between the Output (O) and Normalized Impact (NI) indicators, highlighting that there is no significant linear relationship between them.
Keywords: indicators, SIR Iber 2020, Latin American Universities, research factor 1. INTRODUCTION
University higher education institutions (IES) and their academics are classified according to their ability to produce and publish scientific papers, their link with the productive sector, and their dissemination through scientific journals of recognized academic prestige [1] [2] [3] [4]. The SIR Iber ranking performs this classification with at least one (1) document published in journals indexed in Scopus in a five-year lapse analyzed [5] [6] [7]; considering indicators of impact, excellence, leadership, open access, editorial management, and innovation, in order to encourage academic quality, competitiveness, visibility of universities [5] [8] [9].
The SIR assesses globally the research activity of IESs and, consequently, through the use of bibliometric indicators weight their scientific activity. Data are obtained from scientific documents of any type indexed in the Scopus database [6] [7] [8]. As of 2020, eighteen (18) indicators are considered (distributed in three (3) main factors): research, innovation, and social impact [5], of which twelve (12) belong to the first factor [5] [6].
The purpose of this research is to analyze and compare the twelve (12) indicators of the research measurement factor of LATAM's IESs in the TOP 10 of the SIR Iber 2020 Ranking, in order to contrast the performance of each institution in the production and dissemination of their scientific papers.
Nomenclature
SIR
SIR Iber
SJR
SRG
SCimago Institution Ranking
Ibero-American Ranking of Higher Education Institutions SCImago Journal and Country Rank SCImago Research Group
IES Higher education institution
LATAM Latin America
2. SIR IBER 2020
The SIR Iber 2020 [5] [6] is the twelfth installment of the annual report on the classification of universities according to the number of papers indexed in the Scopus database in the five-year period 2014 - 2018. This report is taken in account to analyze their performance depending on three different sets of indicators based on: research performance, innovation results, and social impact measured by their web visibility.
The SIR Iber 2020 reached the evaluation figure of 1,748 university institutions [5] [6]. There is no mathematical expression that relates the indicators with different weights and that, finally, provides a value of a total indicator per university. The research measurement factor, which constitutes 50% of the composite indicator, is built based on the institutional capacity to generate and disseminate scientific products in scientific communication publications. It includes twelve (12) indicators which could provide twelve (12) different rankings (according to the indicator used) [9] [10]. The report is prepared by the SCImago Research Group and contains these measurements, but the Output (O) indicator (which quantifies the amount of scientific publications of the IES in journals indexed in Scopus [7] [8] during the analyzed five-year lapse) is the one that determines the relative ordering, and it is strengthened with the institutional contribution of said scientific works in relation to excellence, leadership, standardized impact, open access production, and the ability to develop editorial management [7] [8] [9].
For 2020 the Weighted Citation Normalized (WNC) indicator has been included in the research factor: as a complement to the traditional Normalized Impact indicator to show the Normalized Impact of each published document based on the number of thematic categories (to which the journal belongs publication), and both to calculate the expected citation of each category and to calculate the impact of a given set of documents. A WNC indicator higher than 1 reflects an average impact higher than the category of the magazine. While a WNC indicator below 1 indicates an average impact lower than the category of the magazine [5] [6] [10] [11]. According to the capacity to publish scientific works in the Latin American region, Brazil, Spain, Portugal, Mexico, Chile, Argentina, and Colombia repeat as the countries with the highest production capacity: they achieved more than 60,000 jobs published in the period 2014 - 2018. Therefore, they constitute group 1 of analysis for the SIR Iber 2020 report [5] [6] [12]. For this paper, higher education institutions (IESs) located in the top 10 of LATAM's countries have been considered (Table 1): Brazil, Mexico, Chile, Argentina and Colombia, to contrast their research measurement factor indicators.
3. COMPARISON OF RESEARCH MEASUREMENT FACTOR INDICATORS AMONG LATAM'S IESS LOCATED IN THE TOP 10 OF THE SIR IBER 2020
For the Output (O) indicator (Fig. 1) it stands out that all universities are positioned in this top 10 for their ability to publish in scientific journals [5] [6] [12]. USP (Brazil), UNAM (Mexico), and UNESP (Brazil) concentrate 50% of the scientific production in this group of Latin American universities. The three universities (in that order) occupy the first three places in the Latin American ranking.
6% 5% ■ USP
6% ■ UNAM
27% ■ UNESP
7% ■ UNICAMP
8% ■ UFRJ
12%
9% ■ UFRGS
9% 11% UFMG
UBA
Fig. 1. Output (O) indicator of LATAM's IESs located in the top 10 in the SIR Iber 2020 report [5]
Regarding the International Collaboration (IC) indicator (Fig. 2), it stands out that all universities have a capacity of over 29% to create scientific collaboration networks [5] [6] [12]. In first place, UCHILE (Chile) is located with 55.88%. Then there is the UNAM (Mexico) with 42.52%. In third place, there is the UBA (Argentina) with 42.35%.
Fig. 2. International Collaboration (IC) indicator of LATAM's IESs located in the top 10 in the SIR
Iber 2020 report [5]
Alternatively, in relation to the Normalized Impact (NI) and Weighted Citation Normalized (WCN) indicators (the latter new in the SIR Iber 2020 edition), Fig. 3 shows that the UCHILE (Chile) universities repeat in the first place as in the IC indicator with 0.83 and 0.87, correspondingly. In second place is the UNICAMP (Brazil) with 0.80 and 0.84 for the NI and WCN indicators. Finally, the UFRGS (Brazil) reaches the values of 0.78 and 082. This university shares third place with USP (Brazil) in the WCN indicator. Furthermore, when measuring the gap with respect to 1 of the NI and WCN indicators [5] [6] [12], it stands out: that the citation of the production of these universities located in the top 10 in Latin America is below the world average of citation. In addition, they are located in an impact average lower than the category of the magazine, respectively, for achieving values lower than 1. Another result that should be highlighted is the similarity of the data for both indicators for each university, which generates similar curve lines. In addition to the aforementioned, the gap with respect to 1 of the WCN indicator is slightly larger than the gap with respect to 1 of the NI indicator.
Fig. 3. Normalized Impact (NI) and Weighted Citation Normalized (WCN) indicators of LATAM's lESs located in the top 10 in the SIR Iber 2020 report, and gap of the Normalized Impact (NI) and Weighted Citation Normalized (WCN) indicators with respect to 1 [5]
Regarding the High Quality Publications (Q1) indicator (Fig. 4), between 40% and 51% of the works published by these universities are found in journals located in the highest 25% of each category of knowledge (according to the established indicator in the SCImago Journal Rank) [5] [6] [8] [12]. UCHILE (Chile) with 51.13%, then UBA (Argentina) with 49.71%, and; in third place, the UFRJ (Brazil) with 45, 13%.
49, 751, 13
44,! )0 L 5,1 j f \ V
44 "4 V 71 4U, ib 44,4
39, 29 39, 99
Fig. 4. High Quality Publications (Q1) Indicator of LATAM's IESs located in the top 10 in the SIR Iber
2020 report [5]
Then again, in regard to the Excellence (EXC) indicator (Fig. 5), between 8% and 14% of the papers published by these universities are within the 10% of the most cited papers in their respective scientific field [5] [6] [12]. UCHILE (Chile) with 13.86%. Then, there is UNICAMP (Brazil) with 12%. In third place is the UFRJ (Brazil) with 11.07%.
Fig. 5. Excellence (EXC) Indicator of LATAM's lESs located in the top 10 in the SIR Iber 2020 report
[5]
Fig. 6 shows the values of the Scientific Leadership (L) indicator, which indicates between 55% and 65% of the papers published by each university are led by a main researcher belonging to the same [5] [6] [12]. UBA (Argentina) occupies the first place with 64.21%. In second place, UFRGS (Brazil) with 62%. In third place is the UFMG (Brazil) with 59.9%.
Fig. 6. Scientific Leadership (L) indicator of LATAM's lESs located in the top 10 in the SIR Iber 2020
report [5]
Fig. 7 contains the values of the Excellence with Leadership (EwL) indicator for this group of top 10 universities, where UCHILE (Chile) stands out with 5.62%, followed by UNICAMP (Brazil) with 5.11%, and then UFRGS (Brazil) with 4.9%. All IES achieved a percentage between 3.5% and 5.7% of the papers published by an institution whose corresponding author belongs to that institution. In addition, they are within the 10% of the most cited papers in their knowledge category [5] [6] [12].
Fig. 7. Excellence with Leadership (EwL) indicator of LATAM's IESs located in the top 10 in the SIR
Iber 2020 report [5]
Regarding the Scientific Talent Pool (STP) indicator (Fig. 8), it represents the number of different authors belonging to the same university who have participated in the total of published papers. [5] [6] [12]. USP (Brazil) obtained the maximum value of 1. UNESP and UNAM share the second place
with 0.42. Finally, the UFRJ (Brazil) achieved a value of 0.33.
Fig. 8. Scientific Talent Pool (STP) indicator of LATAM's lESs located in the top 10 in the SIR Iber
2020 report [5]
The Open Access (OA) indicator (Fig. 9) (with values between 40% and 62% of scientific papers published in open access journals or indexed in the Unpaywall database [5] [6] [12]) achieves its highest value in UNESP (Brazil) with 61.18%, followed by UNIFESP (Brazil) with 51.57%, and UCHILE (Chile) with 50.61%.
Fig. 9. Open Access (OA) indicator of LATAM's lESs located in the top 10 in the SIR Iber 2020 report
[5]
The Not Own Journals (Not OJ) indicator refers to the number of documents not published in the journals published by the institution itself [5] [6] [12]. USP (Brazil) has 65,434 documents (27%). UNAM (Mexico) follows it with 28,052 (11%). Finally, UNESP (Brazil) has 26,392 (11%) documents. These lESs stand out in the first three places (Fig. 10).
6% A ■ USP ■ UNAM
5%
6% 27% UNESP
7% ■ UN ICAMP
B% 11% ■ UFRJ
9% 10% 11% UFRGS
UFMG
Fig. 10. Not Own Journals (Not OJ) indicator of LATAM's IESs located in the top 10 in the SIR Iber
2020 report [5]
And, lastly, the Own Journals (OJ) indicator, which is defined as the number of journals edited and published by the IES [5] [6] [12], presents the following results: UNAM (Mexico) with 31 (25%), USP (Brazil) with 30 (24%), and UNESP (Brazil) with 13 (11%) own journals. Those previously named occupy the first three places (Fig. 11).
Fig. 11. Own Journals (OJ) indicator of LATAM's IESs located in the top 10 in the SIR Iber 2020
report [5]
Table 2 shows a summary of the position of the universities in the first three places [5]. The following stand out in the first position: UCHILE (Chile) in six (06) indicators: IC, NI, WCN, Q1, EXC, and EwL; USP (Brazil) in three (03) indicators: O, STP, and Not OJ; UBA (Argentina), UNESP (Brazil), and UNAM (Mexico) in one indicator: L, OA, and OJ, individually.
Table 2. Position of LATAM's IESs located in the Top 10 in the SIR IBER 2020 report according to the
indicators of the research measurement factor [5]
Output International Collaboration Normalized Impact Weighted Citation Normalized High Quality Publication s Excellence Scientific Leadership Excellence with Leadership Scientific Talent Pool Open Acess Not Own Journals Own Journals
8% 2% 13% 2% 2% 5% 8% 2% 2% 3% 3%
Number % Normalized 1 % % % % Number % Number Number
Position O IC NI WCN Q1 EXC L EWL STP OA Not OJ OJ
First place USP UCHILE UCHILE UCHILE UCHILE UCHILE UBA UCHILE USP UNESP USP UNAM
Second place UNAM UNAM UNICAMP UNICAMP UBA UNICAMP UFRGS UNICAMP UNAM, UNESP UNIFESP UNAM USP
Third place UNESP UBA UFRGS USP, UFRGS UFRJ UFRJ UFMG UFRGS UFRJ UCHILE UNESP UNESP
4. RELATIONSHIP BETWEEN THE O AND NI INDICATORS AMONG LATAM'S IESS LOCATED IN THE TOP 10 OF THE SIR IBER 2020
Figure 12 shows a scatter diagram between the O and NI indicators according to data taken from the SIR IBER 2020 report [5], where it is shown that there is no significant linear correlation between both indicators. In other words, the fact that a university increases its capacity to publish in scientific journals (O) does not increase the impact of the knowledge generated by it in the international scientific community (NI).
Fig. 12. Correlation between the O and NI indicators according to SIR Iber 2020 report [5]
5. CONCLUSIONS
The SIR Iber ranking classifies universities based on their ability to produce and publish documents in recognized journals in the scientific field. In the 2020 edition, twelve (12) indicators in the research factor stand out by including the Weighted Citation Normalized (WCN). As a result of the analysis and comparison of the indicators considered in this ranking (with respect to LATAM's IESs in the TOP 10), it stands out: Although USP (Brazil) is in first place in Latin America (with the three (03) indicators: O, STP and Not OJ), UCHILE (Chile) stands out in first place in six (06) indicators: IC , NI, WCN, Q1, EXC, and EwL. UBA (Argentina), UNESP (Brazil), and UNAM (Mexico) stand out in the first place in an indicator: L, OA, and OJ, respectively. It was also evidenced that there is no significant linear correlation between the O and NI indicators. That is, the fact that a university increases its capacity to publish in scientific journals (O) does not increase the impact of the knowledge generated by it in the international scientific community (NI).
REFERENCES
[ 1] M. Torres-Samuel, C. Vásquez, M. Luna, A, Viloria and T. Crissien. Eficiencia técnica de la investigación y desarrollo, ciencia y tecnología, educación e innovación en países latinoamericanos. Iberia Journal of Information Systems and Technologies (RISTI). RISTI N° E29, 05/2020. P.P. 582-594. ISSN: 1646-9895 http://www.risti.xyz/issues/ristie29.pdf, 2020.
[2] L. García, M. Riverón and L. López, «Los indicadores como herramienta para evaluar la internacionalización de la investigación de las universidades,» RECUS: Revista Electrónica Cooperación Universidad - Sociedad, pp. 91-99, 2016.
[3] C. Vásquez, M. Torres-Samuel, A. Viloria, T. Crissien, N. Valera, M. Gaitán-Angulo and J. P. Lis, «Visibility of research in universities: the triad producto-researcher-institution. Case: Latin American Contries,» Lectur Notes in Computer Sciencie (Including subseries Lectur Motes in Artificial Intelelligent and Lectur Notes in Bioinformatics), 2018.
[4] C. Vásquez and M. Torres-Samuel, «Public policies in science and technology in latin American countries with universities in the top100 of web ranking,» Journal of engineering and applied science, 2017.
[5] F. De-Moya-Anegón, E. Herrán-Páez, A. Bustos-González, E. Corera-Alvarez, G. Tibaná-Herrera and F. Rivadeneyra. Ranking iberoamericano de instituciones de educación superior 2020 (SIR Iber). Granada: Ediciones Profesionales de la Información. ISBN: 978 84 120239 3 0 https://doi.org/10.3145/sir-iber-2020SCImago, [Last access: 25 06 2020], 2020.
[6] SIR Iber 2020. Informes SCImago. SCImago Institutions Rankings. [On line], Available: https://recyt.fecyt.es/index.php/SCImagoEPI/issue/view/3827/showToc. [Last access: 25 06 2020], 2020.
[7] S. I. Rankings, «SCImago Institutions Rankings,» SCImago Lab & Scopus, 2020. [On line], Available: https://www.scimagoir.com/. [Last access: 08 07 2020].
[8] S. I. Rank, «SJR SCImago Journal & Country Rank,» Scopus, 2020. [On line], Available: https://www.scimagojr.com/. [Last access: 08 07 2020].
[9] C. Vásquez, M. Luna, M. Torres-Samuel, N. Bucci, A. Viloria and T. Crissien, Producción científica y posicionamiento de universidades según ranking SIR IBER 2013 al 2019, caso latinoamericano. Iberia Journal of Information Systems and Technologies (RISTI). RISTI N° E29, 05/2020. P.P. 570-581. ISSN: 1646-9895 http://www.risti.xyz/issues/ristie29.pdf, 2020.
[ 10] M. Torres-Samuel, C. Vásquez, M. Luna, N. Bucci, A. Viloria and D. Cabrera. Clustering of Top 50 Latin American Universities in SIR, QS, ARWU, and Webometrics Rankings. (2019). Procedia Computer Science Volume 160, Pages 467-472. https://www.sciencedirect.com/science/article/pii/S1877050919317636. https://doi.org/10.1016/j.procs.2019.11.063, 2019.
[11] M. Torres-Samuel, C. Vásquez, M. Luna, N. Bucci, A. Viloria and J. Vargas, Characterization of Latin American Universities in the TOP10 of the Rankings. Proceedings of 6th International Conference on Big Data and Cloud Computing Challenges pp 105-118. https://link.springer.com/chapter/10.1007/978-981-32-9889-7_9, 2019.
[12] SCImago, «SCImago Intitute Ranking,» SCImago, 2019. [En línea]. Available: https://www.scimagoir.com/. [Last access: 01 07 2020].
