Economic journal in Russia: quality assessment issues

Due to the continuous increase in the number of scientific journals, it has become very important to assess their quality, which would allow us to understand what publications guarantee a high level of materials they publish. Previously this function was fulfilled, to a certain extent, by the List of leading peer-reviewed scientific journals and editions that are authorized to publish the major research findings of dissertations, the so called VAK List.

In accordance with the order of the Ministry of Education and Science of Russia dated July 25, 2014 No. 793, as amended by the order of the Ministry of Education and Science of Russia dated June 03, 2015 No. 560, a new List of peer-reviewed scientific editions that are authorized to publish the major research findings of candidate’s and doctor’s dissertations (hereinafter – the List)1. The number of journals in it has increased significantly compared to the previous list. However, as noted in the statement of the Council on Science under the RF Ministry of Education and Science dated March 31, 2016, the new list, formed by application principle, included “a large number of publications of a low scientific level that do not ensure a serious professional examination of published materials”, and they publish articles that have “large-scale incorrect borrowings”2. It is emphasized that in Economics, Pedagogy and Law, such journals represent over 15% of all publications in the relevant specialty included in the List. The Council has proposed to conduct gradual replacement of the List with the list of journals of the Russian Science Citation Index (hereinafter – the RSCI list) on the Web of Science platform, taking into account the fact that the quality of such journals is confirmed by the examination carried out in Russia and abroad.

In this paper, we attempt to analyze economic journals included in the List and in the RSCI list, and to assess the appropriateness of possible replacement of the List with the list of journals of the Russian Science Citation Index.

As of April 19, 2016, 388 journals on the specialty “08.00.00 – the Economic Sciences” are on the List. Here it should be noted that among them there are not only economic, but also multidisciplinary editions. General characteristics of these journals may be given by examining the values of their impact factors.

The impact factor was introduced by the American scientist Eugene Garfield and he first used it as a term in the article published in the journal “Science” in 1955 [14]. The impact factor is a basic scientometric indicator of citation. Foreign and Russian experts adhere to the opinion that the impact factor can serve as an estimate of a journal’s authority (Campbell Ph. [13]; Garfield E. [15]; Gordon M.D. [16]; Saha S. [18]; Libkind A.N. [4]; Markusova V.A. [5]; Shaikevich I.V. [6]; Pislyakov V.V. [9]; Rozenberg G.S. [10]). Being essentially an indicator of the authority of articles, this indicator measures the number of citations received on average by one article that is included in a given set of publications for a certain fixed period of time [15]. In our opinion, there is no reason to argue with C. Hoeffel, who indicates that although the impact factor has not become the ideal indicator for measuring the quality of articles, at present, it is the most objective tool for scientific evaluation. Hoeffel thinks that its wide distribution as an indicator of usefulness and influence is due to the fact that it correlates well with the opinion which has been established among scientists about the best journals in their disciplines [17, p. 1225]. As we have noted earlier, when using the impact factor to assess scientific journals, it is necessary to consider the fact that its value may be sufficient to characterize the journals included in authoritative databases (e.g. Web of Science and Scopus) that select their content based on very strict criteria that allow inferior publications to be weeded out [12, p. 163]. When we are dealing with indicators of the impact factor calculated in the Russian

Science Citation Index, the obtainment of more accurate results requires application of additional quantitative characteristics of journals, in particular, the indicators that taking into account self-citation, in order to exclude the issue of artificial overstatement of indicators. Due to this fact, when assessing the journals, we used the value of the RSCI two-year impact factor without self-citations. We ranked all the 388 journals listed in the specialty “08.00.00 — the Economic Sciences” with the use of the above criterion, and determined the proportion of the journals with different values of the impact factor in the total number of publications. The results are presented in Figure 1 .

Our analysis shows that only 19 journals have the impact factor greater than unity. Their share is about 5%. We can say that these editions are highly-cited. These figures indicate that a relatively small number of economic journals publish a significant number of articles that receive a large share of citations. Experts in the field of scientometric studies point out that this is a common trend as among all scientific journals, including those that are in the global system of citation. Thus, according to the analysis carried out on the journals indexed in the Web of Science database, 4% of the journals account for 51% of all citations [7, p. 113]. In the language of scientometric indicators, this is reflected in the high values of impact factors, significantly exceeding the corresponding indicators for other journals.

The impact factor of the majority of economic journals included in the List is less than unity. At that, the share of low-cited

Figure 1. Distribution of the journals included in the List on the specialty “08.00.00 – the Economic Sciences” according to the value of the two-year RSCI impact factor (excluding self-citation) for 2014 (data of the Scientific Electronic Library as of February 20, 2016), %

journals with the impact factor less than 0.3 is 45%. The share of the journals, the impact factor of which is not calculated in the Russian Science Citation Index at all is 9.5%. Thus, it appears that the List includes more than 50% of the journals in the “Economic Sciences” specialty, which have a low level of citation or are virtually not cited at all. And if, in the opinion of experts, the high value of the impact factor does not guarantee the quality of the journal, as this figure may be raised artificially due to the vulnerability of the Russian Science Citation Index to potential manipulation [8, p. 133], then its low value is more revealing: it indirectly indicates that the journal is not in the “mainstream of science” [2, p. 75], its significance is local, availability and attractiveness to the audience are low, and it has no proper scientific credibility.

The above analysis proves that when forming the list of peer-reviewed scientific editions that are authorized to publish the findings of dissertational studies, along with other criteria, it is advisable to use the tools that helps assess the level of the journal. It is very important that the evaluation should have quantitative expression and serve as a specific measure for ranking the journals. One of these tools may be a criterial value of the two-year impact factor, which will help identify the journals with sufficient citation rate. In our view, the median of impact factors can be used as a criterion indicator. In our case, for economic journals, the median is equal to 0.3, i.e. half of the journals has the impact factor above 0.3, while the other half has the impact factor below 0.3. This approach would help clear the List from a significant share of journals with low citation rates and that are not in demand; thus the current list of publications in the specialty “08.00.00 — the Economic Sciences” would be reduced to approximately 170 journals.

Having in mind the imperfection of the new VAK List, the Council on Science under the Ministry of Education and Science of the Russian Federation made a suggestion to the Ministry of Education and Science to replace it gradually by the list of journals of the Russian Science Citation Index (hereinafter — the RSCI list) on the Web of Science platform, and, as before, to include those publications that are covered in international databases and citation systems like Web of Science, Scopus, MathSciNet and others listed in the current Regulations on awarding academic degrees3. The RSCI list was established through evaluation and selection of scientific journals indexed in the Russian Science Citation Index, to improve visibility and enhance the prestige of Russian scientific journals in the international information space. Currently it has just over 650 journals. Journals are selected on the basis of their bibliometric and expert assessment: the list of the journals that meet certain formal criteria is revised by experts in different scientific areas. According to the report of the Working group on the evaluation and selection of journals for this project, the final decision on the inclusion of the journal in the RSCI list was made on the basis of their compliance with formal selection criteria, a number of bibliometric indicators of RSCI and taking into account the results of the expert assessment of the journals in the main thematic areas and public examination by leading Russian scientists4 .

A complete list of journals indexed in the RSCI in the “Economic Sciences” specialty and included in the RSCI list is presented in Table 1 . The list includes 29 titles that were ranked according to the value of the integral index used in constructing the ranking of Russian scientific journals in the Science Index system. The table additionally includes the information about the publisher/founder, the place of publication, and the inclusion of the journal in the VAK List, and their coverage in the global citation indexes Web of Science and Scopus.

Analysis of the values of the two-year impact factor excluding self-citation for these journals (fig. 2) shows that the share of highly cited journals (the impact-factor of which is more than 1) among them is 24%, which is almost five times greater than the corresponding indicator for economic journals included in the new VAK List. The value of the two-year impact factor for more than 70% of the journals is in the range of 0.3—0.9. Thus, it is possible to say that the RSCI list mostly includes the journals with sufficient citation rates, i.e., the journals that have considerable influence and importance in their knowledge field.

Table 1. Bibliometric indicators of the journals in the Economic Sciences covered in the Russian Science Citation Index and included in the database of RSCI database, 2014 (data of the Scientific Electronic Library as of February 20, 2016)

No.	Journal	Founder/publisher	City	Coverage in the List, citation indices	IF2 excluding selfcitation	Integral indicator Science Index
1.	Voprosy ekonomiki	Non-commercial partnership “Editorial of the journal “Voprosy ekonomiki”	Moscow	VAK	4.676	20.458
2.	Rossiiskii zhurnal menedzhmenta	Saint Petersburg State University. Graduate School of Management	Saint Petersburg	VAK	1.000	5.502
3.	Journal of Institutional Studies (“Zhurnal institutsional’nykh issledovanii”)	CJSC “Gumanitarnye perspektivy”	Rostov-on-Don	VAK	1.359	4.231
4.	Forsait	NRU “Higher School of Economics” (Moscow)	Moscow	VAK, Scopus	2.077	3.785
5.	Problemy prognozirovaniya / Studies on Russian Economic Development	Institute of Economic Forecasting, RAS	Moscow	VAK, Scopus	1.419	3.756
6.	Mirovaya ekonomika i mezhdunarodnye otnosheniya	Russian Academy of Sciences (RAS), Institute of world economy and international relations (IMEMO) Russian Academy of Sciences	Moscow	VAK	1.234	3.720
7.	Ekonomicheskii zhurnal Vysshei shkoly ekonomiki	NRU “Higher School of Economics”	Moscow	VAK	0.936	3.198
8.	Innovatsii	OJSC “Transfer”	Saint Petersburg	VAK	0.558	2.606
9.	Zhurnal novoi ekonomicheskoi assotsiatsii	Autonomous Non-Commercial Association “The Journal of the New Economic Association”	Moscow	VAK	0.860	2.068
10.	Ekonomicheskaya nauka sovremennoi Rossii	Central Economic Mathematical Institute of RAS	Moscow	VAK	0.988	2.036
11.	Region: ekonomika i sotsiologiya	Siberian Branch of RAS; Institute of Economics and Industrial Engineering within the Siberian Branch of the Russian Academy of Sciences	Novosibirsk	VAK	1.043	1.811
12.	Den’gi i kredit	Central Bank of the Russian Federation	Moscow	VAK	0.970	1.468
13.	Vestnik Sankt-Peterburgskogo universiteta. Seriya 8: Menedzhment	Saint Petersburg State University. Graduate School of Management	Saint Petersburg	VAK	0.568	1.355
14.	Vestnik Sankt-Peterburgskogo universiteta. Seriya 5: Ekonomika	Saint Petersburg State University	Saint Petersburg	VAK	0.880	1.306

End of the Table 1

No.	Journal	Founder/publisher	City	Coverage in the List, citation indices	IF 2 excluding selfcitation	Integral indicator Science Index
15.	Universitetskoe upravlenie: praktika i analiz	Non-commercial partnership “Journal “ Universitetskoe upravlenie: praktika i analiz”	Yekaterinburg	VAK	(0.963 – 2013)	1.209
16.	Ekonomika i matematicheskie metody	Russian Academy of Sciences; Central Economic Mathematical Institute of RAS; Market Economy Institute of RAS	Moscow	VAK	0.552	1.176
17.	Prikladnaya ekonometrika	Moscow University for Industry and Finance “Synergy”	Moscow	VAK	0.482	1.078
18.	Problemy upravleniya	Sensidat-Plyus	Moscow	VAK	0.538	1.051
19.	Prikladnaya informatika	Moscow University for Industry and Finance “Synergy”	Moscow	VAK	0.600	0.943
20.	Vestnik mezhdunarodnykh organizatsii: obrazovanie, nauka, novaya ekonomika	NRU “Higher School of Economics”	Moscow	VAK	0.537	0.887
21.	Ekonomicheskaya politika	The Russian Presidential Academy of National Economy and Public Administration; Gaidar Institute for Economic Policy	Moscow	VAK	0.915	0.826
22.	Korporativnye finansy	NRU “Higher School of Economics”	Moscow	VAK	0.368	0.826
23.	EKO	Institute of Economics and Industrial Engineering within the Siberian Branch of the Russian Academy of Sciences (Novosibirsk)	Novosibirsk	VAK	0.633	0.678
24.	Vestnik Finansovogo universiteta	Financial University under the Government of the Russian Federation	Moscow	VAK	0.424	0.649
25.	Terra Economicus	Southern Federal University	Rostov-on-Don	VAK WoS (ESCI)	0.584	0.573
26.	Vestnik Moskovskogo universiteta. Seriya 6: Ekonomika	Lomonosov Moscow State University	Moscow	VAK	0.391	0.496
27.	Biznes-informatika	NRU “Higher School of Economics”	Moscow	VAK	0.579	0.458
28.	Upravlencheskie nauki	Financial University under the Government of the Russian Federation	Moscow	VAK	0.506	0.314
29.	Ekonomika. Biznes. Banki	Russian-Italian International University	Moscow	VAK	0.195	0.061

Figure 2. Distribution of economic journals included in the RSCI database according to the value of the two-year impact factor RSCI without self-citation in 2014 (data of the Scientific Electronic Library as of February 20, 2016), %

However, we think that the proposal to replace the VAK List with the RSCI list, initiated by the Council on Science under the Ministry of Education and Science of the Russian Federation, cannot be accepted unconditionally. On the one hand, we cannot but agree with the opinion of many experts who point out that the new List includes many low-quality journals, journals with low citation rates do and journals that have no authority in the scientific community. Obviously, today this list cannot be regarded as an authoritative source able to confirm the proper status of a scientific journal. But on the other hand, the RSCI list in its present form leaves a lot of questions unresolved.

First, according to the statement of the Working group on the evaluation of journals for this project, the journals were selected on the basis of their bibliometric evaluation; after that, the list of journals that meet certain formal criteria was adjusted by experts in different scientific areas5. It is clear that this approach helps exclude all the journals that have low bibliometric indicators at the first stage already. However, the analysis of the group of economic journals alone shows that the final list contains a journal with low citation rate (the two-year impact factor excluding self-citation is 0.195) and a low value of the integral indicator in the Science Index system, which takes into account important parameters characterizing the quality level of scientific publications. In the 2014 Science

Index ranking in the “Economics. Economic Sciences” category, this journal ranks 238 (out of 300 journals; the data of the Scientific Electronic Library as of April 25, 2016).

Second, in our opinion, a big problem of the list of economic journals included in the RSCI, is the fact that their geographical distribution is not extensive (fig. 3) . It includes the journals published mostly in Moscow (20) and Saint Petersburg (4); their share in the total number of economic publications in this list is about 83%. It includes only five journals that are issued in the regions. It should be emphasized that several federal districts are not represented in the RSCI list at all: for example, the Volga, North Caucasian, and Far Eastern federal districts. However, the largest federal districts — the Central and Northwestern — are represented by the scientific journals published exclusively in their central cities.

While the constituent entities of these districts contain a significant number of scientific institutions and research centers that conduct serious research, including the research in the Social Sciences, and publish high-quality journals with high bibliometric indicators and credibility in the academic community. It is obvious that when creating the list of journals recommended for publication of dissertational research findings, it is necessary to take into account the regional aspect: if the leading role belongs to journals issued in central areas, then the economic science in the regions will suffer irreparable loss.

It should be pointed out that the share of the journals of the Russian Academy of Sciences in the RSCI list, in particular the journals on economics, is very modest. Their share is only 20%. Although many experts agree that today the scientific success of the

Figure 3. Geographical distribution of economic journals included in the RSCI list

journal depends largely on its connection to RAS and on its adherence to academic traditions and standards, which are important in the formation of the leading scientific editions of the country [1, p. 111-112]. Scientists believe that academic journals serve as a kind of model for authors and publishers in terms of “structure, clarity of expression and clarity of thought” [3, p. 173]; the fact that a journal is published under the Russian Academy of Sciences indicates the high quality of the publication. And it is not just words. Academic scientific journals were among the first to shift to international publishing standards, they implemented best practices of reviewing manuscripts and provided through these measures the quality of their publications corresponding to a world level. As a result, several journals published by RAS institutions (we emphasize that these institutions are regional), entered global citation indices. For instance, the journal “Ekonomika Regiona” (Institute of Economics, Ural Branch of RAS, Yekaterinburg) is now covered in Scopus; the journal “Economic and Social Changes: Facts, Trends, Forecast” (English version; ISSN: 2307-0331) issued by the Institute of SocioEconomic Development of Territories of the Russian Academy of Sciences (Vologda), is covered in the Emerging Sources Citation Index, which is a new database in the Web of Science Core Collection. But, despite the fact that both journals have high publication performance in the RSCI (tab. 2), enjoy

Table 2. Bibliometric indicators of scientific journals in economics and related disciplines issued by economic institutes under RAS in 2014 (data of the Scientific Electronic Library as of February 20, 2016)

Journal Institute City GO cc о ° E 1 8 ° 1 ? I -cd y= О od ^ ^ § ^ 41 £ а sli| ^ - 8 re У ° 1 re ° |1 У ш re E o ° ° re - 8 csl ш " ° .У о ™ E g re 8 Ekonomicheskie i sotsial’nye peremeny: fakty, tendentsii, prognoz Institute of Socio-Economic Development of Territories of RAS Vologda 1.117 18 15.7 429 48 Prostranstvennaya ekonomika Economic Research Institute, Far-Eastern Branch of RAS Khabarovsk 1.057 20 8.6 411 45 Ekonomika regiona Institute of Economics, Ural Branch of RAS Yekaterinburg 0.960 28 12.2 1090 19 Zhurnal ekonomicheskoi teorii Institute of Economics, Ural Branch of RAS Yekaterinburg 0.670 47 27.6 574 84 Problemy razvitiya territorii Institute of Socio-Economic Development of Territories of RAS Vologda 0.661 49 21.6 732 103 Vestnik Instituta ekonomiki RAN RAS Institute of Economics Moscow 0.500 80 16.4 458 120 authority in the scientific community and have made significant progress in terms of integration into the global academic space, they are not included in the RSCI list. The table also shows the main bibliometric indicators of scientific journals in economics and related disciplines for some economic institutes under RAS. The table presents the journals with a significant average citation rate (the two-year impact factor RSCI without selfcitation is 0.5 and above). According to the basic bibliometric indicators that characterize citation, all of these journals are superior to many publications, included in the RSCI list. Their scientific credibility in professional circles is beyond doubt, because these journals are published by influential economic institutes under the Russian Academy of Sciences, have a strict system of peer review, and publish the findings of basic research conducted by leading economists. Some of these journals have already adopted modern publishing standards and they have high-quality bilingual websites. Despite all this, they were not included in the RSCI list.

We do not set before ourselves the task to carry out a similar analysis for other scientific journals in economics published by universities and other organizations. Probably, among them there are also publications that could complement the RSCI list.

Without questioning the importance of the Russian Science Citation Index project for the promotion of Russian journals and their integration into the international information space, we would like to note that the first results require comprehensive consideration and further research. So far, experts are cautious in their assessments of this project.

Thus, O.V. Kirillova, Director of the Training and Consulting Center at NP NEICON, an authorized consultant and expert for Scopus gives a generally positive assessment of the RSCI and points out that neither variant of the examination takes into account international standards, and as a result, the RSCI list includes even those publications that have no necessary data translated into English, and the publications in which do not meet strict international standards ⁶ . Obviously, all this will not allow such titles to be integrated into the international information space, and will not make them more visible to the global scientific community.

In our opinion, the RSCI, in particular, the list of economic journals, can and should be complemented by quality publications that have, first of all, the full-text English-language version, possess modern publishing platforms, bilingual sites and meet global requirements. This list should include a larger number of academic journals that traditionally have a strict system of peer review and publish the results of fundamental research. And, of course, the geography of the journals needs to be expanded. If the geographical distribution of the journals is narrow, and in the case of economic journals it is limited mainly by Moscow-based editions, this can create significant obstacles to the development of economic science in the regions.

It is clear that the very procedure of evaluation and selection of journals needs to be more transparent and open to the scientific community. This is especially concerns expert evaluation, which in essence can have a certain amount of subjectivity. At least, while the results presented by the authors of the project are assessed positively, there are still the issues concerning the inclusion of some publications in the list and the exclusion of some of them from the list.

Summarizing all of the above, we should note that today, indeed, there exists a difficult situation in the sphere of scientific journals. In our opinion, the new VAK List needs to be revised, because in its present form it is not an authoritative source to confirm the proper status of a scientific journal. The RSCI list, which was proposed as an alternative by the Council on Science under the Ministry of Education and Science of the Russian

Federation and which received cautious assessments by the expert community, requires reflection and further development. In our view, this list, even if it is supplemented by the journals included in international citation indexes, is not yet ready to fully replace the List of peer-reviewed journals that are authorized to publish the major research findings of candidate’s and doctor’s dissertations. However, if the geography of the journals on the RSCI list is expanded, and if this list includes a larger number of academic and other scientific periodicals that meet international standards and have sufficiently high bibliometric indicators, this project has good prospects to obtain wide support in the scientific community as the guarantor of the quality of the scientific journals it contains.