Publishing activity of RAS economic institutes amid new challenges of science policy
Автор: Tretyakova Olga Valentinovna
Журнал: Economic and Social Changes: Facts, Trends, Forecast @volnc-esc-en
Рубрика: Discussion platform
Статья в выпуске: 6 (48) т.9, 2016 года.
Бесплатный доступ
New challenges of science policy in Russia have determined a number of objectives which require scientific institutions to implement specific solutions aimed at improving research quality and efficiency. Since the evaluation of research results requires the use of different metrics, including bibliometrics, the main issue lies in the choice of indicators which can be used to conduct multi-analysis. The aim of this study is to implement the holistic approach to bibliometric assessment of scientific institutions and to analyze the performance of academic activity of economic institutions on the basis of the proposed criteria. The article summarizes the results of scientometric indicators monitoring, conducted at the Institute of Socio-Economic Development of Territories of the Russian Academy of Sciences. The analysis is based on a large set of publications of academic economic institutions for the period 2011-2015. The informational basis of the research includes data of the Russian Science Citation Index (RSCI)...
Science policy, publishing activity, ras economic institutions, performance of academic activity, bibliometric indicators, citation analysis, russian science citation index
Короткий адрес: https://sciup.org/147223892
IDR: 147223892 | DOI: 10.15838/esc.2016.6.48.15
Текст научной статьи Publishing activity of RAS economic institutes amid new challenges of science policy
It is obvious that Russian scientific organizations face new challenges. They are expected to deal successfully with the challenges raised before modern society and our country and to offer effective responses to them. The government relies on the organizations that support and develop relations with the real sector of the economy and social sphere, demonstrate high effectiveness of research and are able to adjust the areas of their work, recognizing the responsibility before the society and the state so as in the future they could achieve fundamentally new and breakthrough results and solve major problems the country is facing.
The issue concerning the allocation of leading scientific organizations again brings to the fore the problem of evaluating their performance and the problem of increasing the indicators, including publication indicators. Scientific organizations must be prepared to fit into a modern system for monitoring and evaluating their performance efficiency and effectiveness. Despite the fact that experts have not yet come to a consensus about the use of publication activity indicators for evaluating the scientific performance of scholars and research teams, scientometric parameters are used as the targets of the state policy in the field of science. The Order of the Ministry of Education and Science of the Russian Federation of March 05, 2014 No. 162 approved the procedure according to which research organizations are to submit the information about the results of their activities and the composition of this information2. The information on the scientific institutions subordinated to the Federal Agency for Scientific Organizations (FANO of Russia) has already been collected. Probably, the results of the monitoring will be announced soon. According to the analysis of the parameters that assess the impact and relevance of scientific research, these parameters include bibliometric indicators, in particular, the number and cumulative citation index of publications of an organization that are indexed in Russian and international information-analytical scientific citation systems, as well as the cumulative impact factor of the journals that contain the articles of this organization. It should be noted that the mandatory data include the number and cumulative citation of the publications indexed in the Web of Science database, while data for other databases are optional. In our opinion, the use of the indicators of global citation indices does not provide a clear picture for evaluating research activity of Russian scholars, since there are not many publications by Russian scientists in international databases (as of May 2014, the share of publications of Russian researchers in scientific journals indexed in WoS amounted to 2.106%). Moreover, the comparison of simple indicators such as the number of published papers and the total number of their citations allows institutions to be ranked according to their overall scientific productivity, but these data are insufficient to identify organizations that carry out breakthrough research.
Consequently, a full-fledged bibliometric evaluation of scientific results requires the use of additional criteria.
In our opinion, a more correct approach to the measurement of the level of scientific productivity of organizations can be based on the analysis of the values of the Hirsch index ( h- index) that reflect a balanced assessment of the publication activities of an organization and citation of its papers. Besides, in order to determine the percentage of higher quality articles in the total publication volume of an organization it is important for the Russian research institutes to consider the number of publications in the journals included in the List of peer-reviewed scientific publications recommended by the Higher Attestation Commission (VAK). Organizations that have a breakthrough performance can be identified with the use of the data on highly cited publications. Thus, the aim of the present work is to try and apply a comprehensive approach to the bibliometric assessment of scientific organizations and to analyze the scientific impact of economic institutes on the basis of selected indicators.
The article summarizes the results of a monitoring of scientometric indicators of scientific organizations for 2011–2015; themonitoringisconductedattheInstitute of Socio-Economic Development of Territories of RAS. The reference group consists of 19 institutions, which are now subordinated to the Federal Agency for
Scientific Organizations of Russia, and were formerly included in the Economics Section, Social Sciences Department (hereinafter – RAS economic institutes). The data for analyzing their publication activity and citation level of their works were taken from the Russian Science Citation Index (RSCI) developed on the basis of the Scientific Electronic Library (SEL, eLibrary.ru). The conclusions presented in the article were obtained in the analysis of a large array of publications that comprises several thousand original documents. Organizations were ranked according to several criteria: total number of publications in the RISC and in the journals on the VAK List and the number of citations. The author provides statistical data on several indicators (in particular, the h- index), the dynamics of which is not reflected in the Russian Science Citation Index. At the same time, the study covers the results of significant analytical work on identifying the highly cited publications of institutions.
The conclusions obtained in the course of the analysis can be useful to study the trends of development of economic science in Russia. In addition, the results of this study are important for the organizations themselves, because they allow them to assess their own performance not only with the help of the dynamics of their own indicators, but also by their comparison with those of other institutions in the relevant reference group.
Methodological approaches to the evaluation of research organizations based on the analysis of bibliometric indicators
Today, a variety of bibliometric indicators are used for monitoring the scientific performance of organizations. The literature on the subject describes various approaches to the analysis of the data obtained on the basis of bibliometrics, but, according to experts [8, p. 108], there are no perfect indicators. The main problem, in our opinion, consists in the definition (choice) of important indicators that can be used to make a multilateral, comprehensive assessment.
For the purposes of carrying out the monitoring of the performance of research organizations simple indicators are used most frequently, in particular, the number of published papers and the number of citations. For example, comparing these indicators over a specific period and in the same region, complemented by rationing them by the number of scientists allows us to rate organizations according to their total research productivity [5, p. 124].
With regard to the evaluation of scientific activity on the total number of publications, as you can see, even those scholars who were at the origins of scientometric studies in our country [6] emphasized that the mere number of publications cannot be considered a criterion of effectiveness of the researcher. But taking the citation rate of the publication as a measure of its usefulness, experts consider it possible to assess the performance of individual scientists and entire research teams on the basis of citation analysis [1, 2, 6]. The notions of citation as an indicator of impact and a tool for assessing scientific contribution stem from theoretical works by R. Merton. He points out that if a scientist’s work goes unnoticed and is not used by other members of the scientific community, then doubts will arise as to its value [22]. Researchers consider the role of citation from different angles, noting that it somehow measures the “intellectual influence” [22, 25], “usefulness and importance of the work” [16], “credibility of the cited document” [18]. According to experts, citation can be used as a tool to measure the impact [14], which the work has on the community as a whole.
As a rule, citation indices are used for evaluating scientific contribution of a researcher, organization, or country as a whole. Earlier, the average citation level was often used for this purpose [3, p. 44]. But today, experts in the field of scientometrics are inclined to believe that average indicators do not give “a complete picture of the researched multitude of papers” and allow for “making a correct comparison of efficiency of research activity of different authors or organizations” [7, p. 2]. Therefore, in recent years, the analytic methods fundamentally departing from the definition of mean values are widely used, in particular, the calculation of “Hirsch-like” indicators [1; 5; 8; 15; 19; 21] and methodology for defining “extreme” citation [3; 4; 8; 12; 23].
The h-index is essentially a measure of scientific productivity of a researcher, based on the distribution of citations of his/her work. This calculation is based on the idea of American physicist J. Hirsch, who in 2005 proposed to set the ratio of the number of a scientist’s publications to the number of their citations. This ratio is called the Hirsch index or the h-index. When introducing this metric, J. Hirsch argued its preference to criteria such as the number of papers divided by the total number of citations, or the number of citations per paper [21]. Despite the fact that the original definition of the h-index focuses on the comparison of the authors’ performance, this indicator can be used for assessing scientific communities. In relation to scientific institutions, as we have mentioned previously [9, p. 196], the h-index is calculated in the RSCI on the basis of the distribution of citations of researchers. The h-index h is obtained if out of the total number (Np) of papers of the staff of the institution each paper is cited at least h times, while the remaining (Np – h) papers are cited no more than h times each. For example, if the h-index of a scientific organization is equal to 10, it means that the database from which the data for the analysis was taken contains not less than 10 scientific works of the institute, each of which has been cited 12 times and more. We can say that the h-index characterizes the magnitude and success of research and publication activities of an organization and reflects the average publication activity of its researchers.
Techniques associated with the definition of the so-called “extreme citation” [8, p. 97-100] focus on the analysis of the articles that have received abnormally low or abnormally high number of links. For the objectives of the present study the author used elements of the methodology for identifying highly cited publications. According to the traditional methodology used in the Essential Science Indicators3 database, the highly cited papers are scientific publications, which at a fixed issuing year and topic were among the 1% of the most cited works4. Treating them as works of the highest quality from the point of view of international recognition of research findings by scientists from a certain country [3; 17; 20; 23; 24], experts conclude that highly cited publications can serve as a kind of indicator of the quality of the scientific system, because in many key fields of science they account for the most significant portion of the links [13]. In the analysis of highly cited articles the absolute rate is investigated, i.e. one finds out how many of such works
Table 1. Dynamics of the indicators of the total number of publications of RAS economic institutes in the Russian Science Citation Index (data of SEL as of November 11, 2016)
In our view, to characterize the performance of research organizations it is important to perform a full-fledged bibliometric evaluation, which involves the combination of several techniques for collecting and analyzing the indicators. For the purposes of the study presented in the paper we used a comprehensive approach. The data obtained in the study of standard indicators such as the total number of publications and citations are supplemented with conclusions on the assessment of the dynamics of values of the h- index of scientific institutions and the analysis of indicators of highly cited publications. The analysis allowed us to characterize the dynamics of publication activity of RAS economic institutes, and to assess the importance of their scientific performance results.
Publication activity of RAS economic institutes in 2011–2015
The total number of publications of RAS economic institutes for 2011–2015 indexed in the RSCI amounted to more than 23.5 thousand, which is 9.5 thousand more than in the previous five-year period; the growth was 169% (in 2010– 2014 there were 14 thousand publications
[10, p. 237]). In this amount, the share of the top five institutions accounts for more than half of the publications – 13.5 thousand (57%): Institute of Economics of the Ural Branch of RAS – 15.2%, RAS Institute of Economics – 11.5%, Institute of Economics and Industrial Engineering of the Siberian Branch of RAS – 11.0%, RAS Institute of SocioEconomic Development of Territories – 9.6%, RAS Central Economics and Mathematics Institute – 9.6%. Dynamics of the indicators of the total number of publications of RAS economic institutes is presented in Table 1 compiled with the use of RSCI data as of November 11, 2016. The institutes are ranked according to the total number of publications during the period under consideration.
As we can see, the peak of publication activity of the institutes falls on 2014 when the average growth rate of the total number of publications amounted to 123.8%. Some institutes have increased the number of publications in 1.5–2 times (Central Economics and Mathematics Institute – in 1.6 times; Economic Research Institute of the Far Eastern Branch of RAS – in 1.6 times; Institute for Social and Economic Research, Ufa Scientific Center of RAS – in 1.8 times; RAS Institute of Agrarian Problems – in 1.9 times; Institute of Socio-Economic and Energy Problems of the North Komi Science Center, Ural Branch of RAS – in 2 times). In our view, the enhancement of publication activity is partly due to the adoption in early 2014 of a new structure of information on the results of activity of scientific organizations and the order of their submission for the purposes of monitoring (Order No. 162 of the Ministry of Education and Science of the Russian Federation dated March 05, 20145). As part of the information, the indicators for evaluating the performance of scientific organizations were established, in particular, the indicators to assess the impactandrelevanceofscientificresearch, including the number and cumulative citation of the publications indexed in Russian and international information-analytical systems for scientific citing. A detailed analysis of the results of activities of scientific organizations, calculated according to information-analytical systems of scientific citation was provided earlier [10].
Returning to the analysis of the growth rate of the number of publications of economic institutes, we should point out that six organizations showed a positive dynamics throughout the whole five-year period (2011–2015) (Fig. 1). It is obvious that a planned increase in publication indicators may prove the systemic efforts undertaken by the institutions in this direction.
For a more detailed assessment of publication activity of scientific organizations, in our opinion, it is necessary to consider additional indicators with which it is possible to analyze the dynamics of publications with respect to editions of different types. Unfortunately, the current data presented in the Russian Science Citation Index and reflecting the number of articles published per year in the journals indexed in Web of Science Core Collection and in Scopus may not be used for analysis. On the one hand, they do not display all the journals in which the institutes publish their papers, but only those descriptions which are contained in the RSCI, and, on the other hand, the system will mark as indexed all the articles from the publications that have been recently included in these databases plus the archives over the years that are not indexed there. All this prevents us from identifying the actual volume of publications in the journals covered in WoS or Scopus on the basis of RISC data alone.
Speaking about the number of publications in different types of scientific editions in relation to the Russian realities, we should pay special attention to the dynamics of the number of articles published by the institutes in the journals included in the List of peer-reviewed scientific editions authorized to publish
Figure 1. Growth rate of the number of publications of RAS economic institutes in the Russian Science Citation Index, % (data of SEL as of November 11, 2016)
|
ooi |
||
|
svy lAiaids |
||
|
zss |
||
|
8 901 |
||
|
7'801 1 |
SVy ЭУ5 |
|
|
Г 511 9'791 ^^^^^B |
||
|
6'88 |
||
|
OVl |
svy ЭУХ 31 |
|
|
9'807 ^^^^^^^^^^^Л |
||
|
9'178 |
||
|
svy 9ПЭ5Х Nd33SI |
||
|
7'97 |
||
|
6'7Vl ■ |
9УУ 53У1 |
|
|
9'96 |
||
|
6 1'101 |
svy dVI |
|
|
6'901 ^^^^^^^^M |
||
|
E01 ■ |
svy dS3SI |
|
|
6711 7'911 ] |
||
|
— |
||
|
о VLSI |
||
|
svy OSS НУ351 |
||
|
A 11791 ° rs0T ww |
svy 93d 1УЗ |
|
|
m 8'891 ■ 06 |
||
|
svy Э50 У351 |
||
|
2 801 I о 8'781 |
||
|
svy 131Л1 |
||
|
™ 87G ■ 801 |
||
|
7'901 |
||
|
901 |
svy Э5ПУ351 |
|
|
9'1701 |
||
|
8'701 |
svy OS» S3I3 |
|
|
9'601 9’701 |
||
|
svy 1Q3SI |
||
|
V 681 7'601 | |
||
|
8'011 |
||
|
8'811 |
svy 331 |
|
|
1'701 9’78 |
||
|
6'691 |
svy 11Л13Э |
|
|
8’76 8'901 |
||
|
9'88 |
||
|
7'971 |
svy as 3i3i |
|
|
V601 976 | |
||
|
8'79 |
||
|
8'76 |
svy 31 |
|
|
6’901 |
||
|
9'601 ^^^^^^^^^ |
||
|
1'071 |
svy an 3i |
|
|
9’901 rzii м |
||
|
о о о о Я ° S 5 |
Ш |
|
main research findings of candidate’s and doctor’s dissertations (hereinafter – the VAK List)6. Traditionally it is believed that such editions guarantee that the level of materials published in them is high. Consequently, the number of “VAK” publications can be considered as a criterion for identifying the proportion of articles of a higher quality in the total volume of publications of an organization.
On the basis of information on the number of articles published in VAK List editions by economic institutes of the Russian Academy of Sciences (presented in Tab. 2), we have found out that the proportion of their publications in the selected editions is 50% of the total amount. The dynamics of indicators on individual organizations is uneven, but for the period from 2011 to 2014 in general it is possible to conclude that there is an annual increase in the number of publications under consideration. However, in 2015, this number decreased almost twofold. The overall decline is due to a substantial reduction in the number of papers published in VAK journals by several institutes. The causes of this decline require independent consideration. In our opinion, it can be caused by a decrease in the volume of research in connection with the reform of the academic system, and by the reorientation of the staff of these institutions on the publication of important results of their research in foreign journals.
Citation analysis
In order to carry out a qualitative estimate of publication activities of RAS economic institutes of the reference group, we used the citation index, which is understood as the amount of links distributed by years to the papers executed in the relevant scientific field.
Table 3 shows the ranking of institutes by the frequency of citation of their works. The table shows that the total number of citations of scientific publications of RAS economic institutes in the RSCI in 2011–2015 amounted to about 140 thousand, about half of them (more than 67 thousand) fall on three organizations: RASCentralEconomicsandMathematics Institute (27.4 thousand), RAS Institute for Economic Forecasting (22 thousand) and RAS Institute of Economics (17.8 thousand). The results of the analysis of the average annual growth rate of the number of citations show that 13 out of the 19 organizations have a positive trend on this indicator throughout the whole five-year period (2011–2015). The total average annual increase in the number of citations ranged from 15 to 28%. Figure 2 presents the data on the organizations in which the average growth rate of the number of citations over the five-year period exceeded 150%.
Table 2. Dynamics of the indicators of the number of articles of RAS economic institutes in the journals included in the List of peer-reviewed scientific editions authorized to publish main research findings of candidate’s and doctor’s dissertations (data of SEL as of November 11, 2016)
|
oo СО LO О ТБ |
CD ей |
CM |
co |
LO |
со |
N |
ОО |
CD |
О |
JZ |
см |
от |
ОТ |
от |
ОО |
CD |
|||||
|
oo |
oo oo CM |
CD CD CO |
CM |
LO CD |
СО CD ОО |
ОО ОО |
СМ от от |
от |
от со |
от |
от |
CD CD CM |
ОО см |
ОО |
от от |
от |
от |
от от |
от |
о см ОО т- |
|
|
ОТ ■'Г ТТ О ч— CD 4—1 Q ° СМ |
LO co |
CM s |
LO |
LO CD CD |
^ |
ОТ |
ОО со |
°? |
об |
CD |
ОО об ОТ |
о |
°? об |
от от |
от |
||||||
|
ОТ ОТ |
CD eg |
CM |
T- |
^ |
co |
LO |
со |
^ |
ОО |
CD |
О |
СМ |
ОО |
О |
об |
от |
от |
ОО |
Г- |
CD |
|
|
00 |
co CM |
co co |
oo |
LO CM |
от |
см ОО |
ОО см |
CD |
ОО |
CD от |
от ОТ |
от от |
CD ОТ |
ОТ от |
см от |
см |
^ |
CD |
ОТ |
СМ се 03 л- |
|
|
^ ОО О Т— CD -•—1 Q ° см ОТ |
co |
co CD |
°? со CD |
ОТ CD |
об ОО |
об |
оо см |
об CD |
ОТ CD СМ |
от CD |
ОТ |
см от |
оо |
о |
от см |
°? ОО |
ОТ СМ |
||||
|
ОТ |
CD |
oo |
T- |
CM |
LO |
со |
ОО |
О |
CD |
СМ |
^ |
ОТ |
J |
от |
от |
Г- |
ОО |
CD |
|||
|
00 |
co CM |
co co co |
co CM CO |
co CM |
СО со |
ОТ |
со |
см см |
^ |
от |
СМ от |
ОТ от |
ОТ ОТ |
CD |
ОО см |
от |
от |
от |
ОТ |
от от 03 от |
|
|
co CM' IT О i— CD 4—i q o' CM CM |
°? CM |
s |
CM CM |
CD |
ОТ |
от со CD |
от CD |
оо со со |
°? об CD |
О |
°? |
об |
О |
оо см |
°? от |
от |
см от |
ОТ |
|||
|
CO ОТ |
CD eg |
oo |
T- |
CM |
^ |
LO |
со |
^ |
ОО |
О |
CD |
ОТ |
см |
см |
J |
от |
от |
ОО |
Г- |
CD |
|
|
00 |
co CM |
co LO co |
LO co |
CM CD |
ОО |
от |
Г- |
со |
см |
от |
ОО |
i^ |
от |
от от |
ОО см |
см |
от |
^ |
от от |
||
|
CM Д—' о 1— ^o CD 4—1 Q GT' CM CM |
oo CM |
^ LO |
CD |
CD CD |
см см ОО |
от со |
ОО от |
от со |
LQ см ОО |
^ от см |
ОО |
см об ОО |
от |
см |
от CD |
CD см |
CD об |
от см |
ОТ |
||
|
CM |
CD eg |
oo |
T- |
CM |
LO |
со |
^ |
ОО |
CD |
CD |
см |
от |
^ |
от |
от |
2 |
CD |
ОО |
|||
|
00 |
oo oo |
CD CO |
co oo CM |
CO |
со со |
со |
см |
£ |
со |
от |
от |
от |
ОО от |
от от |
CD СМ |
от |
Г- |
от |
^ |
ОТ О от |
|
|
CD eg |
CM |
LO |
со |
со |
^ |
О |
см |
^ |
ОО |
CD |
от |
от |
ОТ |
Г- |
от |
CD |
ОО |
||||
|
00 |
oo |
co |
LO CM |
см см |
CD |
со ОО |
CD от |
О |
ОО от |
Г- |
ОО от |
от |
от |
ОТ |
см |
от см |
СМ |
от |
03 1“ |
||
|
03 = |
CD 00 о E CD о Ё о о "ей CO < CC |
CO < CC 'o co "ей ZD oo Ё о о 'о CD 00 |
о Ё о о 'о CD 00 со < CC |
CD CD CD O) Ё "oo о О 8 1 Ш co 'о со CD CD "Е — CD 00 -О -Е со |
О) 00 ей CD о Ё о о 'о CD 00 со < сс |
'о Е _о CD CD о Ё о о о о со 'о CD Ё 00 4—, CD oo Н о со Ё с £ |
8 ей ей о ^ 00 аз со Ё о о Ё со о С *- сс CD ч_ =3 ° 4^ CD 00 — о |
CD 00 Е о о CD ей ^ со < сс |
ей о ей CD 00 CD сс Ё о о ш со ■о < 8 S Е ° ё 5 Ё 8 CD ^ZZ Ё 8 00 о -Е со |
CD ОО ей ей CD 'о CD to ей CD oo СО CD <Г СС сс ° ° 8 2 ш со |
ей CD 00 CD сс 00 о <С ё г ° ° о "ей о 5? СО о ч- СО ° CZ CD СЙ О 00 ±^ со -Е о |
с со сй С jzz СС ° CD *— оо оз 03 Е СС CD Е £ ° Е О CD ей ^5 "ей Е ё ° 00 « £ £ CD — А ей Е Е CD Z3 |
Е 00 аз ОТ Ё о о о о от to CD Е 8 00 4—1 с ^ ОТ о С О сс □_ |
00 аз ОТ Ё о о "ей О О) CD СС о CD 00 от < СС |
-О "ей > CD О) 4—1 CD CD 8 с о оэ с ° о “ о Е £ 2 о -с= о О со От ^ <; о “ с г ° § ° о ел с "ел Е 2 -Е -Е со |
CD сй CD 00 CD сс 8 CD сй ^ оо Ё о о Е CD Е от Е < е 2 -Е Е |
00 Е -Е "о 8 ей О) < to CD 00 ОТ < СС |
ОТ < СС Е CD Л ей CD 00 CD СС о от |
8 о Ё о о о CD 00 — от о) <С б сс и 8 Е Ё Q С CD Е § ОТ ^ |
—1 г о 1— |
|
Table 3. Dynamics of the indicators showing the total number of citations of RAS economic institutes in the Russian Science Citation (data of SEL as of November 11, 2016)
|
ел ей 03^ LT> |
CD |
CM |
ОО |
LO |
со |
Г- |
СТ |
СО |
ст |
см |
га |
LO |
со |
га |
СТ |
||||||
|
СЛ |
Ln CO CM |
co CM CM |
CD |
LO со |
га га CD |
ОО со со |
см ОО |
га |
со га га га |
га га га |
га га |
га со |
га LO СМ |
ст |
LO СО |
со |
см |
га га |
ст га т- |
||
|
LO ^r ТТ о 1— ^о CD -i—1 q ° см CM |
co CD |
°? CM |
об |
in |
°? CD СМ |
со |
об LO |
со |
об |
со СТ см |
ст га см |
°? см |
33 об |
ст см |
со об |
со LO |
СТ СТ |
СМ ем 1“ |
|||
|
Ю |
CD eg |
T- |
CM |
ОО |
LO |
^ |
со |
га |
СТ |
см |
ст |
га |
2 |
LO |
со |
га |
ст |
||||
|
ел |
co |
CM LO LO |
LO |
со со |
СМ LO |
со см |
со LO см |
га га см |
га см |
5г |
см |
LO га га |
LO LO со |
ст СО LO |
ст LO га |
ст см |
га см |
LO |
LO |
ст ст о о |
|
|
^ CO IT о 1— ^o CD -i—1 CD ° CM cm |
CT co |
°? co |
ст со |
^ СМ |
га |
°? |
ОО |
°? СТ |
°? см см см |
ст СТ |
см |
ст об см |
об ст |
СО см |
33 ст |
об см |
ст со см |
СТ |
га об см л- |
||
|
CM |
CD .S3 |
T- |
CM |
ОО |
LO |
со |
^ |
СМ |
га |
СТ |
ст |
^ |
га |
J |
LO |
Г- |
со |
га |
ст |
||
|
ел |
CM co co |
LO |
со LO ОО |
LO со ОО |
СО со ОО |
см CD ОО СМ |
со |
ОО см |
со га см |
га |
см га |
га га |
см LO |
га га |
га см |
га |
см ст |
га |
UD |
га ОО см га |
|
|
co cm' IT o 1— ^o CD -i—1 CD ° CM cm |
cd |
oq о |
со |
О |
Ln см |
°? |
ст со |
СТ |
°? |
°? ст |
см СТ |
га см |
ст об |
33 об га |
СТ га |
зр СТ |
со ст |
S |
ст 1— |
||
|
CO CM |
CD .S3 |
CM |
ОО |
^ |
LO |
СО |
^ |
ст |
га |
Ст |
см |
га |
J |
со |
LO |
Г- |
га |
ст |
|||
|
ел |
CM LO |
LO co |
CD см ОО ОО |
ОО ОО |
со со LO см |
CD СО СМ |
см |
см со |
со LO LO |
со |
LO LO |
см га |
га см |
СТ |
га |
га ст |
га |
со |
S ст ст см |
||
|
CM IT o 1— ^o CD -i—1 CD ° cm cm |
°? |
LO см |
S |
°? со |
СО LO |
со |
LO |
га |
СТ |
ст см |
со со |
со об |
ст см LO |
°? об |
зЗ со га |
см LO |
га |
га |
га ст л- |
||
|
CM CM |
CD .S3 |
T- |
CM |
ОО |
LO |
со |
^ |
ст |
^ |
ст |
га |
см |
2 |
га |
со |
LO |
га |
^ |
ст |
||
|
ел |
CM CD CD |
CM |
см со см |
со со см |
см |
со LO |
га |
га 1-П |
ОО |
LO |
ст LO |
га см га |
LO |
га га |
см |
со |
S |
со |
ст |
ст ст см см |
|
|
CM |
CD .S3 |
CM |
ОО |
LO |
со |
ст |
СТ |
^ |
га |
см |
J |
со |
LO |
га |
га |
Г- |
ст |
||||
|
ел |
CM co |
CM co |
со LO см |
со см |
СТ |
ст |
СТ |
ОО Ln Ln |
см |
СО со см |
LO га |
см см |
га |
ст |
га см |
см |
LO |
см |
Г" ст 1“ |
||
|
03 c |
CD ел о E CD о Ё о о "E co < cc |
CT ел CO CD о Ё о о 'о CD ел со < сс |
о Ё о о 'о CD ел со < сс |
ст CD CD ’ст Ё "ел о О s И ш ей 'о СО CD СО Зз — Ё CD ел -О -Е со |
со < сс 3d со "Е о ел Ё о о 3d CD ел |
CD ел Е о о CD ей ^ со < сс |
3d Е о CD CD о Ё о о о о со 3d CD Ё ел Ч—। CD ел Н о со Ё с £ |
-S3 & 3d ел S3 СО Ё о о о о со 3d CD ел 30 С С о СС -40 |
ел CD СТ ей О ей CD ел CD СС Ё о о ш со ? § ей Ч— _ О ей и_ о 22 о с СО CD ° о CD ОС ОС о оо о -Е со |
Ц СО -S3 о ^ ел S3 СО Ё о о о СР со Ё < ±й СС 1 ° г 03 ё <0 |
CD ел сс$ ей CD 3d CD Зл ей CD ел СО CD <Г СС сс ? ° 8 2 ш со |
ей О ей CD ел CD СС Ё о о ш со Ст < 73 о ё 5 £ ° CD ЦО ЦО о 00 о -Е со |
^ со ей С jo се ° CD *— ел Е CD СС CD Е £ ° Ё О CD ей аз ё Ё ё ° со « £ £ аз — А ей Ё Е CD Z3 |
ел Е со Ё о о л о ’ст CD СС о CD ел СО < СС |
°^ СЛ гг аз О 2 С1_ СД ^ СТ СТ _ аз Е ш ° ? “ ей с О CD IS о С о аз о СТ о Ё СО О О -ч— -СО аз 1С Ё CD ±й S Ё Ё |
ел Е -Е "о л ей ст < 3d аз ел СО < СС |
3d аз ей аз ел аз СС л аз ей ^ ел Ё о о 3d аз Ё СО ел <£ -Е сс |
ей о Ё о о о аз ел СТ Ё со ел <£ аз СС ® 8 с ей « Е СО аз |
со < сс 3d аз Ё ей аз ел аз СС О со |
—1 г о 1— |
|
Figure 2. Growth rate of the number of citations of RAS economic institutes in the Russian Science Citation Index , % (data of SEL as of November 11, 2016)
|
6 SOT 6'OTT ■ 0917 ZE |
svy DUS |
|
SVB IAI3ldS |
|
|
Т 96 ЕТОТ |
|
|
8 SVT | |
|
|
9 90Z |
sva эа>13i |
|
VZST |
|
|
Z'SZT |
sva avi |
|
6 VET 1 Z'98__ |
|
|
Т 6ZT Г |
|
|
Z Z6T |
sva an os» Ndaasi |
|
Z 8ET Е'ЕТТ |
|
|
6 8ZT г |
|
|
90TZ |
sva saai |
|
6ZST |
|
|
sva ass Haasi |
|
|
VZEZ |
|
|
sva 3sn aasi |
|
|
Z STT ■ 8'9VT |
|
|
g 9SZT |
|
|
гм Z TVT |
sva aa=i iaa |
|
EY6 6'TZT |
|
|
S 6'svt гм EWT ■ T'TST |
|
|
sva 3sx san |
|
|
V'TZT |
|
|
3 E ZZZ О TYST гм TTET |
sva 3sa aasi |
|
■ |
|
|
Г8£Т E STT 6Z0T T SOI ■ |
|
|
sva asasi |
|
|
EST |
|
|
ZTET |
sva inasi |
|
S T9T 17'Z9T Л |
|
|
T 9TT |
|
|
EOTT |
sva laiAi |
|
zsst |
|
|
6'6ZT I-------------- |
|
|
VET |
sva an ai |
|
S'VZT 6'VET |
|
|
S ZOT |
|
|
6'VZT |
sva as 3i3i |
|
T ZOT T TOT ■ |
|
|
VEOT |
|
|
60ET |
sva 3i |
|
8ZTT S ZOT |
|
|
E TZT |
|
|
E'ETT |
sva 33i |
|
8Z6 E OTT |
|
|
17'96 |
|
|
6'9TT |
sva 11Л133 |
|
Г80Т V'TTT |
|
|
Й^^СП^Г^Й^нЗ^ |
Judging by the data (Fig. 3) , the total number of publications of the institutions during the study period increased by 30%, while the number of citations of all the works increased twofold: from 19.1 thousand to 40.1 thousand, respectively. Dynamics of citation metrics reflecting “the impact of scientific papers on the scientific community” [2, p. 43] is an indirectevidenceofincreasingimportance of the publications of RAS economic institutes for scientific discipline.
A comprehensive assessment of the scientific contribution and performance oforganizationsbasedonthesimultaneous analysis of the number of publications and their citation can be built by applying the technique for calculating the h-index. We decided against determining the mean citation values, since the averaging can produce quite distorted results caused by the so-called “emissions”, i.e. publications that have received a lot of links.
The results of the study of the dynamics of the values of the Hirsch index for RAS economic institutes are shown in Table 4 . The feature of the data presented lies in the fact that their dynamics is not reflected in the RSCI. Here the results of a four-year monitoring of scientometric indicators are summarized, the monitoring is held at the Institute of Socio-Economic Development of Territories of RAS. All the values are given for the end of the selected year, the two indicators are presented for the current period: for the beginning and the end of 2016.
Figure 3. Dynamics of the indicators of the total numbers of publications and citations of RAS economic institutes in the Russian Science Citation Index, % (data of SEL as of November 11, 2016)
Table 4. Dynamics of the h-index* of RAS economic institutes in the Russian Science Citation Index (data of SEL as of November 11, 2016)
|
№ |
Name of institution |
2013 |
2014 |
2015 |
2016 (February) |
2016 (November) |
2016 to 2013 |
|
1 |
RAS Central Economics and Mathematics Institute |
23 |
48 |
67 |
77 |
98 |
in 4.3 times |
|
2 |
RAS Institute of Economic Forecasting |
22 |
28 |
37 |
40 |
87 |
in 4 times |
|
3 |
Institute of Economics and Industrial Engineering, Siberian Branch of RAS |
21 |
31 |
46 |
51 |
64 |
in 3 times |
|
4 |
RAS Institute of Economics |
23 |
32 |
50 |
52 |
63 |
in 2.7 times |
|
5 |
RAS Market Economy Institute |
12 |
35 |
49 |
51 |
58 |
in 4.8 times |
|
6 |
Institute of Economics, Ural Branch of RAS |
15 |
21 |
32 |
35 |
43 |
in 2.9 times |
|
7 |
RAS Institute of Socio-Economic Studies of Population |
16 |
21 |
29 |
30 |
36 |
in 2.3 times |
|
8 |
RAS Institute of Socio-Economic Development of Territories |
12 |
14 |
25 |
28 |
35 |
in 2.9 times |
|
9 |
Economic Research Institute of the Far Eastern Branch of RAS |
10 |
14 |
21 |
25 |
34 |
in 3.4 times |
|
10 |
Institute of Social and Economic Research, Dagestan Scientific Center of RAS |
4 |
10 |
21 |
24 |
30 |
in 7.5 times |
|
11 |
Institute for Social and Economic Research, Ufa Scientific Center of RAS |
4 |
9 |
18 |
19 |
28 |
in 7 times |
|
12 |
Luzin Institute for Economic Studies, Kola Scientific Center of RAS |
10 |
12 |
18 |
21 |
27 |
in 2.7 times |
|
13 |
Institute for Socio-Economic and Energy Problems of the North, Komi Scientific Center, Ural Branch of RAS |
6 |
9 |
12 |
13 |
22 |
in 3.7 times |
|
14 |
Institute for Social and Economic Research and Humanities, Southern Scientific Center of RAS |
3 |
7 |
14 |
15 |
20 |
in 6.7 times |
|
15 |
RAS Institute for Regional Economic Studies |
6 |
8 |
12 |
13 |
17 |
in 2.8 times |
|
16 |
RAS Institute of Agrarian Problems |
5 |
9 |
12 |
13 |
14 |
in 2.8 times |
|
17 |
Institute of Economics, Karelian Research Center of RAS |
4 |
8 |
11 |
11 |
14 |
in 3.5 times |
|
18 |
Saint Petersburg Institute for Economics and Mathematics, RAS |
7 |
7 |
9 |
9 |
10 |
in 1.4 times |
|
19 |
Sochi Research Center of RAS |
4 |
5 |
6 |
6 |
7 |
in 1.8 times |
* The h -index is calculated on the basis of the distribution of citations of the publications of the organization and has a value N, if an organization has N articles, each of which was cited at least N times, and the rest of its articles are cited no more than N times. All the types of the citing and cited publications are taken into account.
The data as of November 11, 2016 show that in the period under consideration the values of the Hirsch index increased manifold at all academic institutions. It is logical that the greatest growth rate is observed in the organizations that had low rates at the beginning of the measurements. On average, the indexes increased in 2–3 times. This correlates with the growth rate of the total number of publications and their citations. The highest value of the index is observed in the Central Economics and Mathematics Institute – 98. This means that the total number of publications of the institute researchers, each of which has almost 100 links and more, is also close to one hundred. In general, ten institutes have the h-index that exceeds 30. Such values of the index show that the organizations fairly quickly accumulate the publications that receive many citations.
Highly cited publications of RAS economic institutes
The study of the citation data for RAS economic institutes has shown that many of them have achieved significant results in this direction. In our view, their performance indicators are due to their highly cited publications.
As we have noted, in the terminology of the Essential Science Indicators international database, the works are considered highly cited if they fall within the 1% of the most cited publications in the world among those published in the same year in the same research field. Experts believe such works to be of the highest quality and to have left their mark not just in “science” but in the “science of the higher level” [8, p. 99]. We believe that the basic principles of the methodology for determining highly cited publications can be applied not only to scientific fields but also to certain groups of scientific organizations. Thus, to determine how many highly cited publications a RAS economic institute has, it is necessary to compare its publications to those of other similar organizations. Thus, the selection of highly cited publications of a scientific institute depends not on its own works and their citation, but on how its performance is viewed against the background of all the other organizations in the reference group.
The analysis is carried out on a large array of publications of RAS economic institutes over the period from 2011 to 2015 covering more than 23.5 thousand of original documents. Since it is incorrect to compare the works of different years due to the fact that some of them could get more citations due to the fact that they were published earlier, the comparison is made in several subsets of publications united by the same year of publication. Publications in each subset are arranged in descending order of citation; after that, the upper section covering 1% of the total number is defined, and a threshold value for the number of citations is set out for a publication to get into the selected “top”. These very publications will be recognized as highly cited in this reference group. At the same time a threshold value is established for the number of citations a publication must receive to get into the upper section, and on its basis the number of highly cited papers is calculated for all organizations.
The use of the described methodology allowed us to identify highly cited publications of 236 RAS economic institutes over the period from 2011 to 2015. Figure 4 presents the data on the number of publications for each year of the period indicated. It also displays the threshold values of the number of citations that a work published during the specified year has to receive in order to join the 1% of the highly cited works.
The results of the analysis show that the total number of references to the publications included in the upper section of the most cited works is 14.7 thousand. Thus, the 1% of all publications of RAS economic institutes for the period under
Figure 4. Distribution of highly cited publications of RAS economic institutes for the period from 2011 to 2015 (data of SEL as of November 11, 2016)
□ Number of highly cited publications
—•— Threshold value of the number of citations that a publication must receive in order to join the 1% of highly cited works
consideration accounts for 10.5% of the citations, i.e. every tenth link is made to a highly cited article.
The distribution of the number of highly cited publications by scientific institutions is presented in Table 5 . It has been found out that half of that number is distributed among three organizations: RAS Institute of Market Problems (18.2%), RAS Central Economics and Mathematics Institute (17.4%) and RAS Institute of Economics (14.8%).
The analysis of relative indicators that show the percentage of highly cited articles published by an organization has shown that the largest share of highly cited publications in the total number of all works belongs to RAS Market Economy Institute. We can conclude that, on average, every twentieth publication by its researchers becomes highly cited compared to the publications of other institu- tions within this reference group. Approximately one in 55 publications becomes highly cited among the publications of RAS Central Economics and Mathematics Institute, and one in 60 publications becomes highly cited among those of RAS Institute of Economic Forecasting and RAS Institute of Socio-Economic Studies of Population. Consequently, the presence of a large number of such papers allows an organization to have relatively high citation indices. Thus, on the basis of the data about publication activities of RAS Institute of Socio-Economic Studies of Population, in 2014, three of its papers accounted for 77% of all the links received during the year (560 of 726 citations), and in 2015, four of its papers received 956 links (74% of the total number of citations per year). In all the cases we are talking about the articles prepared by large teams of authors and published in
Table 5. Distribution of highly cited publications of RAS economic institutes in the Russian Science Citation Index (data of SEL as of November 11, 2016)
It is obvious that the analysis of highly cited publications presents additional opportunities for a more detailed assessment of the impact of research works. In our opinion, these data push the limits of the approaches based on the use of absolute indicators to count the number of publications and citations, these indicators are currently used at the official level to monitor and evaluate the performance of research organizations. But their wide application requires the presence of databases that would automatically generate all the necessary data not only about the publications covered in Web of Science, but also at the national level for Russian scientific works.
Conclusion
The present work contains a comprehensive analysis of the publication activity and research impact of 19 economic institutes. They are ranked according to several criteria, in particular, according to the total number of publications in the Russian Science Citation Index and in the journals on the VAK List, according to the number of citations, to the value of the Hirsch index and the number of publications recognized as highly cited for this reference group. The analysis is performed on a large array of bibliometric indicators for the period of 2011–2015, the data having been obtained from the information-analytical system of the Russian Citation Index.
The results of the study prove that scientific organizations are increasing their publication activity, they publish the works important for their scientific specialty and enjoying high demand in the scientific community, as evidenced by continuously increasing citation indicators. The total number of publications of the institutions in the analyzed period has increased by 30%, while the number of citations of all the works has grown twofold. The peak of publication activity of the institutions falls on 2014, which, in our opinion, is partly due to the official approval of the list of information about the results of activities of scientific organizations to be submitted to the monitoring, and which identify the indicators to assess the relevance of scientific research, including the number and cumulative citation of publications indexed in Russian and international information-analytical systems.
A disturbing trend is associated with the fact that in 2015 there was a significant reduction in the volume of publications in the journals included in the VAK List. Since the number of VAK publications is often regarded as a criterion for identifying the proportion of articles of a higher quality in the total amount of publications of an organization, the reasons for the decline in the number of such publications require that research institutes consider them separately and take additional efforts to correct their publication policy.
Nevertheless, the overall increase in citation indicators shows that the results of the work of RAS academic institutes are important for the Russian economic scientific community. The dynamics of the values of the h- index is indirectly confirmed by the increased level of scientific productivity of organizations on average in 2–3 times. This correlates with the growth rate of the total number of publications and links to them.
As follows from the data of citation analysis, significant indicators of several scientific organizations are due to their highly cited publications. In general, 1% of all publications of RAS economic institutes for the period of 2011–2015 account for 10.5% of citations, i.e. every tenth link accounts for the article that is highly cited for this reference group. A study of the relative indicators showing what percentage of the organization’s publications falls within the 1% of highly cited papers has allowed us to establish that on average one in twenty publications becomes highly cited in the organization leading by this indicator.
In conclusion, we note that the presented results enable a more comprehensive approach to the performance evaluation of scientific organizations, but even this approach, in our opinion, does not allow making any management decisions based only on the account of the publication indicators. Comprehensive evaluation requires the participation of experts. Bibliometric information in this case can only serve as an additional source of detailed information about a particular scientific organization.
Список литературы Publishing activity of RAS economic institutes amid new challenges of science policy
- Bredikhin S.V., Kuznetsov A.Yu., Shcherbakova N.G. Analiz tsitirovaniya v bibliometrii . Novosibirsk: IVMiMG SO RAN, NEIKON, 2013.
- Garfield E. Mozhno li vyyavlyat' i otsenivat' nauchnye dostizheniya i nauchnuyu produktivnost'? . Vestnik AN SSSR , 1982, no. 6, pp. 42-50.
- Kotsemir M.N. Dinamika rossiiskoi i mirovoi nauki skvoz' prizmu mezhdunarodnykh publikatsii . Forsait , 2012, volume 6, no. 1, pp. 38-59.
- Markusova V.A., Ivanov V.V., Varshavskii A.E. Bibliometricheskie pokazateli rossiiskoi nauki i RAN (1997-2007) . Vestnik RAN , 2009, volume 9, no. 6, pp. 483-491.
- Moskaleva O.V., Akoev M.A., Markusova V.A., Pislyakov V.V. Nauchnye publikatsii kak sredstvo kommunikatsii, analiza i otsenki nauchnoi deyatel'nosti . Rukovodstvo po naukometrii: indikatory razvitiya nauki i tekhnologii: monografiya . Ekaterinburg, 2014. Pp. 110-163.
- Nalimov V.V., Mul'chenko Z.M. Naukometriya. Izuchenie razvitiya nauki kak informatsionnogo protsessa . Moskva, 1969. 192 p.
- Pislyakov V.V. Shedevry nauchnogo tvorchestva: analiz vysokotsitiruemykh statei rossiiskikh uchenykh . Nauchno-tekhnicheskaya informatsiya. Ser.2: Informatsionnye protsessy i sistemy , 2011, no. 12, pp. 1-8.
- Pislyakov V.V., Akoev M.A., Markusova V.A., Moskaleva O.V. Bibliometricheskie indikatory v resursakh Tomson Reuters . Rukovodstvo po naukometrii: indikatory razvitiya nauki i tekhnologii: monografiya . Ekaterinburg, 2014. Pp. 75-109.
- Tret'yakova O.V., Kabakova E.A. Vozmozhnosti i perspektivy ispol'zovaniya indeksov tsitirovaniya v otsenke rezul'tatov deyatel'nosti nauchnogo uchrezhdeniya . Ekonomicheskie i sotsial'nye peremeny: fakty, tendentsii, prognoz , 2013, no. 6, pp. 189-200.
- Tret'yakova O.V. Razvitie natsional'nogo indeksa tsitirovaniya kak uslovie formirovaniya sistemy otsenki rezul'tatov nauchnoi deyatel'nosti Ekonomicheskie i sotsial'nye peremeny: fakty, tendentsii, prognoz , 2015, no. 1, pp. 230-245 DOI: 10.15838/esc/2015.1.37.15
- Adler R., Ewing J., Taylor P. Citation statistics. Statistical Sciences, 2009, volume 24, pp. 1-14.
- Aksnes D.W. Characteristics of highly cited papers. Research Evalution, 2003, volume 12, no.3, pp. 159-170.
- Aksnes D.W., Sivertsen G. The effect of highly cited papers on national citation indicators. Scientometrics, 2004, volume 59, no 2, pp. 213-224 DOI: 10.1023/B:SCIE.0000018529.58334.eb
- Cole S., Cole J. R. Scientific output and recognition. A study in the operation of the reward system in science.
- American Sociological Review, 1967, volume 32, pp. 377-390.
- Egghe L. Characteristic scores and scales based on h-type indices. Journal of Informetrics, 2010, volume 4, no. 1, pp. 14-22 DOI: 10.1016/j.joi.2009.06.001
- Garfield E. Citation Indexing-Its Theory and Application in Science, Technology and Humanities. New York: Wiley, 1979. 274 р.
- Garfield E. The 100 most-cited papers ever and how we select citation classics. Current Contents, 1984, no. 27, June 4th, pp. 3-9.
- Gilbert G. N. Referencing as persuasion. Social Studies of Science, 1977, volume 7, no. 1, pp. 113-122.
- Glänzel W. On the h-index -A mathematical approach to a new measure of publication activity and citation impact. Scientometrics, 2006, volume 67, no.2, pp. 315-321.
- Glänzel W., Schubert A. Some facts and figures on highly cited papers in the sciences, 1981-1985.
- Scientometrics, 1992, volume 25, pp. 373-380 DOI: 10.1007/BF02016926
- Hirsch J.E. An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 2005, volume 102, no. 46, pp. 16569-16572 DOI: 10.1073/pnas.0507655102
- Merton R. K. The sociology of science: An episodic memoir. The sociology of science in Europe. Carbondale: Southern Illinois Univ. Press, 1977, pp. 3-141.
- Plomp R. The highly cited papers of professors as an indicator of a research group's scientific performance.
- Scientometrics, 1994, volume 29, pp. 377-393 DOI: 10.1007/BF02033446
- Tijssen R.J.W., Visser M.S., van Leeuwen T.N. (2002) Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics, volume 54, no 3, pp. 381-397 DOI: 10.1023/A:1016082432660
- Zuckerman H. Citation analysis and the complex problem of intellectual influence. Scientometrics, 1987, volume 12, no. 5-6, pp. 329-338.
