Opportunities for and prospects of using citation indices in evaluating the performance of research institution

Poor representation of the Russian periodicals in foreign publications, the difficulty in using foreign databases for statistical analysis, as well as their price affordability primarily caused the necessity of introducing the national system for evaluating and analyzing the publication activity and citation of national researchers, organizations, and editions was caused [15].

The bibliographic abstract database that accumulates more than 4.7 million publications of Russian authors and information on the citations of these publications in more than 4 thousand Russian journals forms the basis of the national system. Recently, other types of scientific publications have been also included in RSCI: conference reports, monographs, textbooks, patents, theses. The base contains information about the imprint, subject area, the author of the publication, his/her workplace, as well as keywords, abstracts and itemized reference lists [15].

The creation of systematically expanding common electronic abstract database of scientific articles published in Russian journals, and monographs was supported by the scientific community. However, when it comes to using this database for evaluating the publication activity of a particular academic or scientific organization, the objectivity of RSCI data is disputed11.

In particular, the experts point out that some RSCI indexable publications cannot be unambiguously considered scientific and indicate the presence of algorithmic errors leading to the inconsistency of numeric parameters, incorrect reference of publications to the authors and organizations, loss of publications and references. Self-citation record is another issue of concern. It is noted that the indicators generated in RSCI, are highly dependent on the organizational activities of the directors of research institutions and publishers, timely inclusion of materials in the database. These complaints are justified. But one cannot but see that resources are actively accumulated in RSCI, the experience of similar foreign bases is adopted, and a search for ways to cooperate with foreign institutions of scientometric measurements has been conducted.

According to the authors, the indicators of Russian science citation index can be used for the evaluation characteristics of the activities of certain scientists and institutions as a whole. However, caution must be exercised not to overlook the specifics of scientific activities, and to use these data as a supplement to the qualified expert evaluation.

Taking this fact into account, the authors applied RSCI indicators for evaluating the publication activity of the Institute of SocioEconomic Development of Territories of the Russian Academy of Sciences in comparison with the research institutes included in the Economic Section of the Social Sciences Department of RAS, registered in RSCI, as well as the Vologda Oblast scientific organizations. This approach provides an opportunity to assess the effectiveness of the organization’s research activities in the relevant scientific field and to denote its contribution to the development of the region’s scientific potential.

However, the authors note that the results, presented in the study, do not claim to be exhaustive. They deal with the issues that can be attributed to the problem of the formation of common approaches to the given analysis.

The primary analysis is based on the premise that if science is regarded as a process of acquiring knowledge, it is necessary, first of all, to trace the growth of the number of scientific publications, considering them to be new knowledge bearers. In this aspect, the publication represents a key element in the production of scientific knowledge: it forms research results, makes them public, ensuring their transfer and confirmation.

It was decided to use for our analysis the retrospective indicators since 2008, when RSCI reached proportions sufficient for the analysis and received de facto recognition in the scientific community and official sources.

As follows from table 1 , the total number of the publications of the institutes of the Economic Section of SSD RAS, registered in RSCI for the 2008–2012 period, made up 9958, more than half of them (61.68%) falls on the share of five institutes: RAS Institute of Economics – 19.87%, the Institute of Economics and Industrial Engineering of the Siberian Branch of the RAS – 13.60%, the Institute of Economics, the Ural Branch of RAS – 13.18%, the Central Economics and Mathematics Institute of RAS – 7.84% and the Institute of Socio-Economic Development of Territories (ISEDT) of RAS – 7.19%.

For the period under review the staff of the Institute of Socio-Economic Development of Territories of RAS provided a significant increase in the absolute number of articles – from 108 in 2008 to 207 in 2012, i.e. almost twice. The share of ISEDT RAS in the total number of publications of the institutions of RAS Economic Section, presented in RSCI, grew up to 7% for this period.

It is obvious that the number of publications can only be the primary criteria for evaluating the scientific and publication activity of the

Table 1. Total number of publications of the scientific institutions of the Economic Section of SSD RAS, registered in RSCI (as of October 25, 2013)

Institution Total number of publications in 2008–2012 Share in the number of the publications of the Economic Section of SSD RAS, % Number of publications Total number of publications for 2013 as of October 25, 2013 2008 2012 Growth rate, % RAS Institute of Economics 1979 19.87 406 409 101 224 Institute of Economics and Industrial Engineering, the Siberian Branch of RAS 1354 13.60 267 305 114 166 Institute of Economics, the Ural Branch of RAS 1312 13.18 244 308 126 158 Central Economics and Mathematics Institute of RAS 781 7.84 179 177 99 142 Institute of Socio-Economic Development of Territories of RAS 716 7.19 108 207 192 147 Institute of Economic Forecasting, RAS 646 6.49 123 125 102 20 G.P. Luzin Institute of Economic Problems of Kola Scientific Centre of RAS 644 6.47 107 139 130 53 Institute of Socio-Economic Studies of Dagestan Scientific Centre of RAS 480 4.82 108 55 51 18 Institute of Agrarian Problems RAS 365 3.67 63 72 114 21 Institute of Social and Economic Studies of Population at the Russian Academy of Sciences 352 3.53 69 72 104 48 Institute of Problems of Regional Economy of RAS 287 2.88 52 64 123 26 Economic Research Institute FEB RAS 257 2.58 41 64 156 29 Market Economy Institute of RAS 188 1.89 23 50 217 31 Institute of Economic Studies of Karelian Research Centre of RAS 128 1.29 11 51 464 21 Institute of Socio-Economic and Energy Problems of the North, Komi Scientific Centre of the Ural Branch of RAS 126 1.27 17 25 147 16 Institute of Socio-Economic and Humanities Research of the Southern Scientific Centre of RAS 116 1.16 2 51 25.5 times 27 Institute of Social and Economic Research, Ufa Scientific Centre of RAS 85 0.85 2 28 14 times 25 Saint Petersburg Institute for Economics and Mathematics of RAS 73 0.73 15 9 60 3 Sochi Scientific Research Centre of RAS 69 0.69 17 15 88 7 Total 9958 100 1854 2226 12.2 1182 institution. This indicator does not carry information about the quality of the scientific output and its relevance. Therefore, the citation index is used in modern scientometrics as the most objective indicator, when evaluating the scientific activities. It is understood as the number of references to the works of a particular researcher or organization, carried out in the relevant field of scientific activities, distributed by the years. Citation shows that these works had an impact on other scientists. Table 2 shows rank distribution of the institutions of the Economic Section of SSD RAS according to the citation frequency of papers, done by the research workers over the specified period.

As follows from the table data, in 2008 the total number of citations of ISEDT RAS lagged significantly behind the leading institutes. In subsequent years, this index increased (12 times in 2012, as compared to 2008) that ultimately enabled ISEDT RAS to take the 7th place in the rating of the institutions of the Economic Section of SSD RAS according to the total number of citations for the 2008–2012 period. At the same time, for some obvious reasons, institutes, located in Moscow, have the highest rank. However, the possibility of “non-capital” institutes to rise their rank increases, as it is these institutes that are to participate in the elaboration of measures to modernize the country’s economic space. The trend that took shape in the last two or three years demonstrates the significant potential for the accelerated advancement of the “non-capital” academic institutions along the path.

According to the authors, RSCI solves positively another important task that is connected with the introduction of the indicator of the researcher’s scientific productivity, based on the distribution of citations received by the researcher. This calculation is based on the idea of the American physicist Jorge Hirsch, who in 2005 suggested counting the ratio of the number of the researcher’s publications and the number of citations of these articles12. This ratio is commonly referred to as Hirsch index or h-index.

J. Hirsch preferred the introduction of the given metrics to the number of publications divided by the total number of citations, or the number of citations per one paper [Hirsch, 2005]. Western colleagues of J. Hirsch, noting the advantages of the h-index over other such indicators, emphasize the fact that this index considers not only the number of publications, but their state of being relevant, thus being the result of a balance between the number of publications and the number of citations received by each publication [Glanzel, 2006].

Despite the fact that some scientists expressed doubts about the correctness of the h-index application for calculating the scientific productivity [Lehnmann S., Jackson A.D., Lautrup B.E., 2005; Sidiropuolos, 2007; etc.], noting primarily the inability of this indicator to identify important works made in the past and papers, which continue significantly influencing the scientific thinking. However, one cannot deny that the h-index is useful as the accepted standard of academic achievements.

The original definition of the Hirsch-index is focused on the comparison of the authors’ productivity; nevertheless, it is used for evaluating academic communities.

For research institutions the Hirsch index is calculated in RSCI on the basis of the distribution of citations received by the researcher’s publications. H-index h is obtained, when n of the total number (Np) of the articles of the institution’s research workers has at least h citations each, and the other papers (Np – h) have no more than h citations each. It is possible to say that the Hirsch index characterizes the scale and efficiency of the organization’s research and publication activities and reflects the average publication activity of researchers.

Table 2. Total number of citations of the scientific institutions of the Economic Section of SSD RAS, registered in RSCI (as of October 25, 2013)

Institution	Total number of citations for 2008–2012	Rank	Total number of citations of authors
			2008	2012	Growth rate, %	For 2013 as of October 25, 2013
RAS Institute of Economics	2 655	1	381	702	184	455
Central Economics and Mathematics Institute of RAS	2 389	2	275	721	262	679
Institute of Economics and Industrial Engineering, the Siberian Branch of RAS	2 138	3	213	655	308	363
Institute of Economic Forecasting, RAS	2 032	4	326	495	152	206
Institute of Economics, the Ural Branch of RAS	1448	5	93	646	7 times	483
Institute of Social and Economic Studies of Population at the Russian Academy of Sciences	806	6	93	202	217	84
Institute of Socio-Economic Development of Territories of RAS	677	7	29	333	12 times	327
Economic Research Institute FEB RAS	399	8	27	148	5.5 times	52
Market Economy Institute of RAS	300	9	26	91	350	35
G.P. Luzin Institute of Economic Problems of Kola Scientific Centre of RAS	197	10	16	83	5 times	56
Institute of Problems of Regional Economy of RAS	149	11	8	65	8 times	29
Institute of Agrarian Problems RAS	135	12	8	50	6 times	12
Saint Petersburg Institute for Economics and Mathematics of RAS	119	13	15	26	173	12
Institute of Socio-Economic Studies of Dagestan Scientific Centre of RAS	108	14	11	44	4 times	22
Institute of Socio-Economic and Energy Problems of the North, Komi Scientific Centre of the Ural Branch of RAS	100	15	9	41	4.6 times	17
Sochi Scientific Research Centre of RAS	70	16	4	8	200	12
Institute of Economic Studies of Karelian Research Centre of RAS	62	17	1	37	37 times	18
Institute of Social and Economic Research, Ufa Scientific Centre of RAS	42	18	4	24	6 times	14
Institute of Socio-Economic and Humanities Research of the Southern Scientific Centre of RAS	15	19	0	7	-	13

Table 3 presents the calculated RSCI ranking position of the institutes of the Economic Section of SSD RAS according to the Hirsch index.

Three of the Institutes of the Economic Section of SSD RAS have high values of the Hirsch index in RSCI (from 20 and above): RAS Institute of Economics (23), Institute of Economic Forecasting, RAS (22), Central Economics and Mathematics Institute of RAS (21).

Five institutions demonstrate the average value of the Hirsch index (10 to 20); eleven – less than 10.

The Hirsch index of the Institute of SocioEconomic Development of Territories of RAS makes up 12. The given indicator implies that RSCI database contains not less than 12

research papers of the Institute, each of which has been cited 12 times or more. Such value of the Hirsch-index (12) is accepted in scientometric national practice as the stability indicator of the quality of the scientific output of the research team.

As has been noted above, the citation indexes can be used for evaluating the organization’s contribution to the development of the region’s scientific potential. In this case data on the publication activity is presented as the information about the need for the researcher’s activities and the significance of the results of their activities.

The study of the data presented in RSCI enabled the authors to determine the ISEDT RAS position with regard to the publication

Table 3. Hirsch index by the institutions of the Economic Section of SSD RAS (as of October 25, 2013)

Institution	h-index	Rank position
RAS Institute of Economics	23	1
Institute of Economic Forecasting, RAS	22	2
Central Economics and Mathematics Institute of RAS	21	3
Institute of Social and Economic Studies of Population at the Russian Academy of Sciences	16	4
Institute of Economics and Industrial Engineering, the Siberian Branch of RAS	15	5
Institute of Economics, the Ural Branch of RAS	14	6
Institute of Socio-Economic Development of Territories of RAS	12	7
Market Economy Institute of RAS	10	8
Economic Research Institute FEB RAS	9	9
G.P. Luzin Institute of Economic Problems of Kola Scientific Centre of RAS	7	10
Institute of Problems of Regional Economy of RAS	6	11-13
Saint Petersburg Institute for Economics and Mathematics of RAS	6	11-13
Institute of Socio-Economic and Energy Problems of the North, Komi Scientific Centre of the Ural Branch of RAS	6	11-13
Institute of Agrarian Problems RAS	5	14
Institute of Economic Studies of Karelian Research Centre of RAS	4	15-18
Institute of Socio-Economic Studies of Dagestan Scientific Centre of RAS	4	15-18
Institute of Social and Economic Research, Ufa Scientific Centre of RAS	4	15-18
Sochi Scientific Research Centre of RAS	4	15-18
Institute of Socio-Economic and Humanities Research of the Southern Scientific Centre of RAS	3	19

Table 4. Main scientometric indicators of the Vologda Oblast state research organizations and higher education institutes in RSCI

Institution Publications for 2008–2013 Citation indicator for 2008–2013 Average number of publications per 1 author Average number of citations per 1 author Hirsch index Number Rank position Number Rank position Number Rank position Number Rank position Value Rank position Cherepovets State University 1203 1 398 3 5.76 6 1.40 3 8 3-4 Institute of Socio-Economic Development of Territories of RAS 863 2 1004 1 6.12 5 2.59 2 12 1 Vologda State Teachers’ Training University 650 3 356 4 4.42 7 0.90 6 10 2 Vologda State Technical University 643 4 453 2 3.74 8 0.97 5 8 3-4 Vologda Institute of Law and Economics 399 5 73 5 11.10 2 1.10 4 4 5 Vologda State Dairy Farming Academy named after N.V. Vereshchagin 317 6 70 6 8.71 4 0.63 7 3 6 North-West Research Institute of Milk and Grass Farming of RAAS 58 7 25 8 3.05 9 0.50 8-9 2 7-9 Vologda branch of the Saint Petersburg State University of Economics 57 8 0 12-14 9.00 3 0.00 11-12 0 12-13 Vologda Institute of Business 41 9 1 11 11.50 1 0.50 8-9 1 10-11 Cherepovets Military Engineering Institute of Radioelectronics 20 10 11 9 1.83 12 0.42 10 2 7-9 Vologda Institute of Education Development 19 11 2 10 - - - - 1 10-11 Vologda Oblast Universal Scientific Library named after V.I. Babushkin 11 12 0 12-14 2.50 10-11 0.00 11-12 0 12-13 Vologda Oblast Hygiene and Epidemiology Centre 2 13 33 7 2.50 10-11 10.00 1 2 7-9 Vologda Branch of State Research Institute of Lake and River Fisheries 0 14 0 12-14 - - - - - - activity, as compared with other research institutions and institutions of higher professional education, located on the territory of the Vologda Oblast. It should be noted that by the number of the research organizations the Vologda Oblast is exceeded by the Arkhangelsk and Murmansk oblasts, the republics of Karelia and Komi, Leningrad Oblast, let alone Saint Petersburg, included in the Northwestern Federal District.

Yet the data, reflected in RSCI resources, makes it possible to state that ISEDT RAS has reached the leading position among the Vologda Oblast institutions of science and higher education by the main scientometric parameters ( tab. 4 ).

Thus, the results of the conducted analysis enable the authors to make the following conclusions.

First of all, the study of the RSCI accumulated resource confirms the importance of scientometric indicators for the contemporary scientist individually or for scientific organizations as a whole. This is also dictated by the system of science management, evolving in Russia.

Secondly, the emergence and development of the Russian-speaking resource that is RSCI, stimulates the interest of the Russian scientists, supervisors and specialists to use it in order to increase the efficiency of research activities.

Thirdly, the effectiveness and the efficiency of the activities of individual research organizations can be evaluated with applying the data of the national citation index. Thus, these data suggest that the Institute of SocioEconomic Development of Territories of RAS has been increasing its publication activity, improving the quality of publications, their scientific and practical significance.

Fourthly, RSCI resources have been used inactively yet. In this regard, additional organizational and economic measures are required. This applies, in particular, to enhancing the professionalism of specialists, involved in the formation of the Russian Science Citation Index, the level of database storage, the frequency analysis of the published knowledge results.

The development and comparison of scien-tometric measurements open new dimensions in scientific activities, provide an opportunity to adjust the direction and content of the activities, compare their results with the results of other studies.

However, it should be taken into account that the information presented both in RSCI and foreign citation indexes cannot be used for the absolute evaluation of the scientific activities of individual scientists and institutions as a whole.

These indicators, considering the specifics of scientific activities not always objectively, can be applied along with qualified expert assessment. Being dynamic, the indicators change quite rapidly, making the estimate, based on these data, objective only for a specific time period.

Nevertheless, the results of the analysis, conducted by the authors, with regard to the publication activity of scientific institutions indicate that the work associated with the registration of the publications in RSCI information and analysis system, is extremely important. At present, it is obviously required to present the results of the institutions’ research activities in RSCI more widely, in order to describe and assess objectively and fully the scientific potential of the regions and the country as a whole.