Development of national citation index as a condition for the formation of a system to evaluate scientific research performance

Nowadays, efficient decision-making in the management of science requires creation of a system for objective assessment of scientific research performance, which helps to allocate public funds to promising research areas, the results of which are demanded by the country’s society and economy.

Development of scientometric indicators and their comparison provide new opportunities for research performance evaluation, help to adjust its direction and content and compare research results. Citation analysis helps to identify regularities and to forecast the probable rate of development of science; it is an effective method for studying communication in the professional community and it represents the disciplinary structure of science.

Despite the fact that various aspects of the use of bibliometric indicators for the assessment of research performance are described by foreign scientists (E. Garfield [23, 24], M. Graber [24], J. Hirsch [25], R. Ketzler [28] and others), and domestic researchers (B.I. Bednyi [1], S.V. Bredikhin [2], V.A. Varshavskii [4, 5, 6], O.V. Kirillova [12], V.A. Markusova [4, 13], O.V. Mikhailov [14], V.V. Pislyakov [17] and others), there are no clear evaluation criteria based on bibliometric assessment; there remains a need for a number of scientometric studies to identify sources of information and methods of its extraction and to identify the tasks that can be solved if the necessary information is available.

The goal of this work is to substantiate the necessity to develop a national information-analytical system of citation, which should become part of the system for assessing the effectiveness and performance of Russian scientists and scientific organizations. The results of the monitoring of indicators of publication activity of scientific economic organizations subordinate to FANO of Russia show that the RSCI has accumulated a certain resource, which in the future can be used as a tool to evaluate the performance of research institutions, and as an alternative to international databases, which today are officially used for these purposes, although they do not make a comprehensive assessment of the results of scientific activity of the Russian scientific organizations and individual scientists because of their low representation in global citation indices.

At present, scientific publications and the media are discussing whether it is expedient to use scientometric indicators in the regulation of scientists’ performance. Experts believe that “in connection with the reform of RAS, academic institutions will be subject to certification, and bibliometric indicators (number of publications, citation, impact factor of scientific journals, Hirsch index) will be used as indicators of performance efficiency of research teams” [7]. L. E. Mindeli points out that the transition of leading countries to knowledge-based economy opens up new opportunities for scientometric developments and, in his opinion, “the methodological arsenal of scientometrics will go beyond science itself and will find large-scale application in the monitoring interaction between economic and social actors in the field of knowledge” [7].

Along with scientists who recognize the role of bibliometric indicators as a tool for assessing the relevance of research results, many experts question the assessment of the academic value of research findings that is based on citation data; they argue that citation indicators give a limited and incomplete view of the quality of research1.

Some of the researchers who consider it useful to apply scientometric methods in assessing research performance efficiency think that bibliometric indicators should be used as auxiliary tools, only as a supplement to expert assessment2. It should also be noted that, like foreign scientists3, Russian experts admit that the effectiveness or productivity of research work is advisable to be assessed by the indicator of specific domestic expenditures on academic or basic science at purchasing power parity per article [32].

Although experts have not yet reached a unanimous opinion on the use of publication activity indicators for evaluating research performance of scientists and research teams, scientometric parameters are used as the targets of the state policy in the field of science. In this case the data from foreign systems of citation are taken as a basis.

Thus, for the further improvement of the state policy in the field of education and science and for the training of qualified specialists with regard to innovation economy requirements, the President ordered the Government to “ensure that by 2015 the proportion of publications by Russian researchers in the total number of publications in international scientific journals indexed in the WEB of Science database will have reached 2.44 percent”4.

The Order of the Ministry of Education and Science of the Russian Federation of March 5, 2014 No. 162 has established the procedure, according to which scientific organizations provide the information on their performance results for the purposes of monitoring; the Order has also established the composition of these data [15]. The composition of the data is determined by 25 indicators, which evaluate research performance of scientific organizations in four areas: performance effectiveness and relevance of research; human resources development; integration into the world scientific community; dissemination of scientific knowledge and enhancement of prestige of science; resource support of scientific organizations.

The first group of indicators to assess the effectiveness and relevance of research includes eight indicators, three of which are bibliometric. These include the total number of citations of publications of an organization, indexed in the Russian and international information-analytical science citation systems, as well as a cumulative impact factor of the journals, in which the articles of this organization are published. It should be noted that the number and total citation of publications indexed in the Web of Science database are mandatory indicators, while the data for other databases are optional.

The data of international information-analytical science citation systems have two more indicators by which the degree of integration of an institution in the global scientific community is assessed. These include the number of articles prepared in collaboration with foreign organizations, and the number of scientific conferences with international participation. These indicators take into account the works published in the journals indexed in Web of Science and Scopus, and also scientific conferences and symposia that were followed by publication of the materials indexed in the specified international information-analytical systems of science citation.

Table 1 shows the indicators assessing the research performance of scientific organizations; the indicators are determined according to the data of information-analytical systems of science citation.

However, the global indicators of citation indices do not provide an objective evaluation of scientific performance of Russian scientists, since the number of their publications in international databases is insufficient (as of August 2013, the share of Russian scientists in the total number of publications in Scopus is 1.9% and in Web of Science – 1.2%) [12, p. 13]. It should also be noted that the citation of Russian authors is low. For instance, the citation impact indicator, which is calculated by InCites5 as the average number of citations, expressed as a proportion of the world average (equal to 1), only in recent years, has been approaching the value of 0.5, i.e., two times below the world average indicator. Foreign experts note that the citation impact indicator for Russian publications is one of the lowest among the countries of Eastern Europe, despite the general opinion that science in Russia is very strong6.

Thus, in our opinion, the use of indicators of global information-analytical science citation systems for monitoring and evaluating the performance of scientific organizations in Russia does not provide an objective overview of the publication activity of Russian scientists and research teams.

Table 1. Composition of information on the performance of scientific organizations assessed according to the data of information-analytical systems of science citation

Performance efficiency and relevance of scientific research	The number of publications of an organization, which are indexed in the Russian and international information-analytical systems of science citation:		All peer-reviewed publications during the reporting period (articles, reviews, theses of reports, conference proceedings) are taken into account
	a	Web of Science	The organization provides a searching field as additional information
	b	Scopus	Provided at the discretion of the organization
	c	Russian Science Citation Index	Provided at the discretion of the organization
	d	Google Scholar	Provided at the discretion of the organization
	e	ERIH (European Reference Index for the Humanities)	Provided at the discretion of the organization
	f	Specialized information-analytical system	Provided at the discretion of the organization in case the publications are indexed in an international system
	Total citation of organization’s publications, which are indexed in the Russian and international information-analytical science citation systems:		The total citation of publications of the organization is taken into account
	а	Web of Science	The indicator is calculated automatically using the Search option
	b	Scopus	Provided at the discretion of the organization
	c	Google Scholar	Provided at the discretion of the organization
	d	Russian Science Citation Index	Provided at the discretion of the organization
	Cumulative impact factor of the journals in which the articles of the organization are published		Calculated according to the values of impact factors of journals relevant to the report year
Integration into the global scientific community, dissemination of scientific knowledge, enhancement of prestige of science	The number of articles prepared in collaboration with foreign organizations		Works published in the journals indexed in the Web of Science and Scopus international information-analytical systems of science citation are taken into account
	The number of scientific conferences with international participation, held by the organization		Only scientific conferences and symposia that were followed by publication of materials indexed in the Web of Science and Scopus iinternational information-analytical systems of science citation are taken into consideration

Source: Prikaz Ministerstva obrazovaniya i nauki Rossiiskoi Federatsii ot 5 marta 2014 goda № 162. Prilozhenie 3 “Sostav svedenii o rezul’tatakh deyatel’nosti nauchnykh organizatsii, vypolnyayushchikh nauchno-issledovatel’skie, opytno-konstruktorskie i tekhnologicheskie raboty grazhdanskogo naznacheniya, predstavlyaemykh v tselyakh monitoringa i otsenki” [The Order of the Ministry of Education and Science of the Russian Federation of March 5, 2014 No. 162. Appendix 3 “The Composition of Information about the Performance Results of Scientific Organizations Carrying Out Scientific Research, Experimental Design and Technological Works of Civil Designation Provided for the Purposes of Monitoring and Evaluation]. Available at:

In addition, leading scientists point out that “substitution of national abstract and citation databases with foreign citation systems leads to the fact that Russian scientists orient their scientific research mainly toward the interests of the foreign scientific community that has its own needs for scientific information, which very often do not coincide with what domestic science requires due to the historical realities” [11, p. 11]. The result is “the loss of sovereignty of Russian science in the choice of research areas and the outflow of promising young scientists who have focused on foreign research issues” [11, p. 11].

Thus, an issue concerning the creation of national citation index that could be an alternative to foreign databases is becoming more acute. Global experience shows that the development of national bibliographic databases enhances the level of national scientific publications and also promotes a comprehensive and accurate assessment of the performance of research teams on the basis of their published works.

National abstract databases, such as China Scientific and Technical Papers and Citations7 and Chinese Science Citation Database (China)8, Citation Database for Japanese Papers (Japan)9, Brazil’s National

Database of Research and Science R e sum e s (Brazil), have made significant progress in promoting and demonstrating scientific achievements of their countries.

The issue concerning the creation of an objective system of scientific institutions performance was raised many times, and also at the highest level. For instance, at the meeting of the Council under the President of the Russian Federation for Science and Education in Gatchina (April 30, 2013) V.V. Putin pointed out that “the current tool for evaluating the performance of scientific organizations and the results of their work has certain disadvantages. It does not allow us to single out true leaders in a given research sector, and impact assessments are not linked to funding received by research organisations”.

The President highlighted the necessity to develop a national system for objective assessment of scientific organizations’ performance”, which, in his opinion, “will allow us to concentrate public resources in those areas where we can indeed expect impressive new results, and receive the promising outcomes much needed by our society and economy”10.

In our opinion, the Russian Science Citation Index can become part of such a system. Established in 2005 on a platform of the Scientific Electronic Library (www. , RSCI is a powerful analytical system for assessing scientific performance;

a set of scientometric indicators is calculated and continuously updated in this system.

The national system is based on the abstract and citation database, which contains more than seven million publications of Russian authors and citations to these publications from more than 4.5 thousand Russian journals.

In addition to publications from scientific journals, the RSCI includes conference papers, monographs, textbooks, patents and dissertations. The database contains information about the output data, authors, their affiliation, key words and subject areas, as well as abstracts and reference lists. Based on the objective data, the RSCI helps evaluate research performance efficiency and study in detail the statistics of the publication activity of more than 600 thousand Russian scientists and 1 1 thousand scientific organizations in all the areas of knowledge11.

Previously, we have attempted to present some general approaches to assessing performance efficiency of research organizations on the basis of statistical data provided by the Russian Science Citation Index [20, 21]. The study of the resources accumulated in the RSCI has substantiated the importance of scientometric indicators for the modern scientist in particular and for the scientific community in general.

A monitoring of publication activity indicators is carried out on the example of a group of economic academic institutions subordinate to the Federal Agency of

Scientific Organizations and previously included in the Economics Section of the Department of Social Sciences of RAS. The analysis of scientometric indicators shows that these organizations are increasing their publication activity, improving the quality of publications and their scientific and practical relevance, as evidenced by the steadily increasing citation rates.

Judging by the results obtained, we can say that if in 2013 the RSCI resources were used insufficiently in the specified reference group, then at the beginning of 2015 the majority of the institutions significantly increased their presence in this database.

Due to the fact that the indicators in the RSCI are periodically updated not only for the current year, but also for previous periods, we compared several indicators for the five-year period in order to assess growth rates of publication activity of the institutions of the specified reference group.

In order to ensure comparability of the periods under comparison in the dynamics we used cumulative indicators of the two consecutive five-year periods: 2008–2012 (according to the RSCI data for October 2013)12; and 2009–2013 (according to the RSCI data for October 2014)13, i.e., in a subsequent period we excluded the indicators of the base year of the previous five-year period and added the indicators of the next year. This approach provides for equal time segments, and takes into account the activization of data input.

The indicators for the last five-year period presented in the RSCI, i.e. for 2010–2014, are given for reference. These data are not used for comparison, because they are not yet fully represented in the RSCI. These data identify future trends in development and are used for ranking the scientific institutions.

Table 2 shows the dynamics of the total number of publications in the RSCI by research institutions engaged in economics and subordinate to FANO of Russia. The organizations are ranked by total number of publications for 2010–2014.

The data indicate that the average growth rate of the number of publications for the period 2009–2013 in this reference group was 125% compared with the previous period of 2008–2012. The data as of January 14, 2015 also show the 13% growth in the number of publications for 2010–2014 in comparison with the previous five-year period. Obviously, these numbers will somewhat increase by the end of 2015.

Table 3 shows the dynamics of indicators of the total number of the RSCI citations of research institutions engaged in economics and subordinate to FANO of Russia. Organizations are ranked by total number of citations for 2010–2014.

Citation analysis shows that all the economic institutions subordinate to FANO of Russia have significantly improved their citation metrics for 2009–2013 in comparison with the previous five-year period. An average growth rate for this indicator in the reference group for 2014 amounted to 209%, i.e. the total number of citations has increased more than twice.

The study of the dynamics of the Hirsch¹⁴ index for scientific institutions engaged in economics and subordinate to FANO of Russia proves that this indicator for many institutions has increased significantly in the course of 2014 (tab. 4) .

Thus, the analysis of indicators of publication activity of academic institutions engaged in economic research and subordinate to FANO of Russia shows that in the course of 2014 the indicators of publication activity of these organizations in the RSCI have increased significantly. Substantial growth of bibliometric indicators is caused by the fact that scientific institutions presented the results of their scientific work in the RSCI most widely, which confirms the growing interest in analytical resource. It is obvious that today the Russian Science Citation Index has excellent prospects to become part of a system for evaluating scientific organizations’ performance.

We think that along with professional expert assessment that takes into account research specifics, scientometric indicators presented in the RSCI can form an objective overview of the efficiency of scientific institutions’ performance by different reference groups.

Table 2. Five-year dynamics of the indicators of the total number of publications by research institutions engaged in economics and subordinate to FANO of Russia, in the RSCI

Institution name	Total number of publications for 20082012 (data as of October 2013)	Total number of publications for 2009-2013 (data as of October 2014)	Growth rate 2009-2013 to 20082012, %	Total number of publications for 2010-2014 (data as of January 14, 2015)	Position in the ranking for 2010-2014 (data as of January 14, 2015)
RAS Institute of Economics	1979	2182	110	2324	1
Institute of Economics and Organization of Industrial Production, Siberian Branch of RAS	1354	1744	128	1982	2
Institute of Economics, Ural Branch of RAS	1312	1449	110	1467	3
Central Economic Mathematical Institute of RAS	781	945	120	1459	4
Institute of Socio-Economic Development of Territories of RAS	716	922	150	1201	5
G.P. Luzin Institute of Economic Problems of Kola Scientific Centre of RAS	644	1096	170	1170	6
Institute of Social and Economic Research of Ufa Science Centre of RAS	85	514	в 6 раз	714	7
Institute of Economic Forecasting of RAS	646	678	104	643	8
Market Economy Institute of RAS	188	406	216	464	9
Economic Research Institute, Far Eastern Branch of RAS	257	365	142	457	10
Institute of Social and Economic Research, Dagestan Scientific Center of RAS	480	472	98	421	11
Institute of Agrarian Problems of RAS	365	371	102	382	12
Institute of Social and Economic Studies of Population at RAS	352	372	106	360	13
Institute of Regional Economy of RAS	287	284	99	359	14
Institute of Social, Economic and Humanitarian Studies Southern Scientific Center of RAS	116	185	159	242	15
Institute of Economics of Karelian Scientific Centre of RAS	128	163	127	178	16
Institute of Socio-Economic and Energy Problems of the North Komi Scientific Centre, Ural Branch of RAS	126	163	129	171	17
Sochi Research Center of RAS	69	106	154	99	18
Saint Petersburg Institute for Economics and Mathematics of RAS	73	80	110	68	19
Total	9958	12497	125	14161	-

Table 3. Five-year dynamics of the indicators of the total number of citations for research institutions engaged in economics and subordinate to FANO of Russia, in the RSCI

Institution name	Total number of citations for 2008-2012 (data as of October 2013)	Total number of citations for 2009-2013 (data as of October 2014)	Growth rate 2009-2013 to 2008-2012, %	Total number of citations for 2010-2014 (data as of January 14, 2015)	Position in the ranking for 2010-2014 (data as of January 14, 2015)
Central Economic Mathematical Institute of RAS	2389	4536	190	10223	1
RAS Institute of Economics	2655	5059	191	6400	2
Institute of Economics and Organization of Industrial Production, Siberian Branch of RAS	2138	5141	240	6137	3
Institute of Economics, Ural Branch of RAS	1448	2790	193	3856	4
Market Economy Institute of RAS	300	2504	in 8.4 times	3519	5
Institute of Economic Forecasting of RAS	2032	2876	142	3092	6
Institute of Socio-Economic Development of Territories of RAS	677	1335	197	2027	7
Institute of Social and Economic Studies of Population at RAS	806	1283	159	1508	8
G.P. Luzin Institute of Economic Problems of Kola Scientific Centre of RAS	197	861	in 4.4 times	1116	9
Economic Research Institute, Far Eastern Branch of RAS	399	820	205	1075	10
Institute of Social and Economic Research, Dagestan Scientific Center of RAS	108	352	326	676	11
Institute of Social and Economic Research of Ufa Science Centre of RAS	42	254	in 6 times	385	12
Institute of Regional Economy of RAS	149	225	151	379	13
Institute of Agrarian Problems of RAS	135	228	169	314	14
Institute of Economics of Karelian Scientific Centre of RAS	62	156	252	273	15
Institute of Socio-Economic and Energy Problems of the North Komi Scientific Centre, Ural Branch of RAS	100	169	169	249	16
Institute of Social, Economic and Humanitarian Studies Southern Scientific Center of RAS	15	103	in 6.9 times	222	17
Saint Petersburg Institute for Economics and Mathematics of RAS	119	170	143	198	18
Sochi Research Center of RAS	70	90	129	112	19
Total	13841	28952	209	41761	-

Table 4. Dynamics of the Hirsch index for scientific institutions engaged in economics and subordinate to FANO of Russia, in the RSCI

Institution name	h-index (data as of October 2013)	h-index (data as of October 2014)	Growth rate of the Hirsch index in 2014 as compared to 2013	h-index (data as of January 14, 2015)	Position in the ranking (data as of January 14, 2015)
Central Economic Mathematical Institute of RAS	21	32	152	48	1
Market Economy Institute of RAS	10	34	340	35	2
RAS Institute of Economics	23	29	126	32	3
Institute of Economics and Organization of Industrial Production, Siberian Branch of RAS	15	29	193	31	4
Institute of Economic Forecasting of RAS	22	27	123	28	5
Institute of Economics, Ural Branch of RAS	14	20	143	21	6-7
Institute of Social and Economic Studies of Population at RAS	16	20	125	21	6-7
Institute of Socio-Economic Development of Territories of RAS	12	13	108	14	8-9
Economic Research Institute, Far Eastern Branch of RAS	9	13	144	14	8-9
G.P. Luzin Institute of Economic Problems of Kola Scientific Centre of RAS	7	11	157	12	10
Institute of Social and Economic Research, Dagestan Scientific Center of RAS	4	9	225	10	11
Institute of Social and Economic Research of Ufa Science Centre of RAS	4	8	200	9	12-14
Institute of Agrarian Problems of RAS	5	8	160	9	12-14
Institute of Socio-Economic and Energy Problems of the North Komi Scientific Centre, Ural Branch of the RAS	6	7	117	9	12-14
Institute of Economics of Karelian Scientific Centre of RAS	4	7	175	8	15-16
Institute of Regional Economy of RAS	6	7	117	8	15-16
Saint Petersburg Institute for Economics and Mathematics of RAS	6	7	117	7	17-18
Institute of Social, Economic and Humanitarian Studies Southern Scientific Center of RAS	3	5	167	7	17-18
Sochi Research Center of RAS	4	5	125	5	19

However, it should be noted that the development of a Russian-language resource, such as the RSCI, requires additional organizational and economic efforts.

First of all, the government should adopt a decision that the Russian science citation index should be given the status of a national database. It is important to bear in mind that, if we are talking about assessing the effectiveness and efficiency of performance of Russian scientists and scientific organizations, and the ranking of scientific journals on the basis of any scientometric indicators, it is necessary to take the data of the national citation index as the basis. The development of a national abstract and citation database will largely contribute to the preservation of the sovereignty of the Russian science; it will also make it possible to be guided by mainly national interests in the choice of scientific development priorities.

It is advisable to specify “the number of publications of the organization, indexed in the RSCI” as part of the information about scientific organizations’ performance, submitted to FANO of Russia for the purposes of monitoring and evaluation, as a compulsory indicator of

the effectiveness and relevance of scientific research. In our opinion, the number of publications included in international databases cannot be used as the main scientometric indicator to assess the performance of a Russian scientific organization, since there are not many Russian publications represented in foreign databases, and this indicator does not provide objective overview of the publication activity of research institutions.

It should be noted that the formation of the national citation index must be arranged through the selection of periodicals and individual publications according to strict criteria that would ensure the quality of the materials contained in the database; it would make their assessment according to scientometric indicators more objective.

In conclusion it should be emphasized that the main objective of the system for evaluating research performance should consists in finding the ways for the promotion of scientific institutions, the establishment of a basis for making effective management decisions in the field of science, all this will help raise the level of Russian scientific publications and achieve significant results in all the areas of scientific research.