Ranking of scholarly journals of economic institutes of the Russian Academy of Sciences

Today there are some hundreds of economic journals in Russia. Over a hundred are included in the List of peer-reviewed scientific editions of the Higher Attestation Commission. About 300 publications are indexed in the Russian Science Citation Index in the category “Economics. Economic sciences”, where they can be ranked by impact factor value and other parameters. However, scientists believe that neither the list of VAK nor the impact factor of the Russian Science Citation Index [2, 8] can measure the quality of a scientific journal.

Leading experts are trying to solve the problem of journals to differentiate and select by ranging them according to various indicators. So, the Russian journals in economics are assessed by several ratings, with Murav’ev’s rating (2013) [7, 8], the rating conducted by the HSE Office of Academic Expertise (2015) [11] and Balatskii-Ekimova’s rating (2015) [2] being most famous. The formation of the above classifications is based on different methodological approaches –use of bibliometric indicators, expert judgement or both of them. However, none of these approaches is perfect.

In our opinion, the ranking of journals in economics and related disciplines, compiled by A.A. Murav’ev, is most profound [7, 8]. It includes analyzed bibliometric information. This classification outstands with its use, besides standard criteria, such as an impact factor, of the indicators, specially calculated by the author on the basis of references published in journals. Fourteen indicators are considered; they are divided in five generalized grades by means of standardization and aggregation. The conducted work results in the selection of 24 leading journals in economics and related disciplines, of which the ten are attributed to the “A” conventional category (most important) and the rest to the “B” category (journals of a good level). Weaknesses of the proposed method, according to its author, is manifested in the imperfection of bibliometric indicators used to reflect scientific merit of journals and the neglect of multidisciplinary journals. We believe there is another disadvantage of this approach: the greater part of analyzed statistical information is not available in the RSCI and the significant resources are required for its generation. It hinders constant updating of the rating and assessment of the ranking results by a broad audience.

The project implemented by the HSE Office of Academic Expertise involves expert assessments as well [11]. Having filled in the questionnaire, 630 experts assessed the scientific quality of journals in 13 areas of science, including economics. Aggregation of individual assessments helped identify 3 conditional groups of journals: A1 – high level, “wide profile”, A2 – high level, “narrow profile”, B – average level. Two journals of group A2 in economics and 17 journals of group B are only singled out. It is interesting to note that “Foresight” and “The HSE Economic Journal” are the journals included in the category A2, i.e. the publications affiliated with the organization, composing the rating, are the best among economic journals. It is noteworthy that in other lists of journals and the RSCI rating these publications do not occupy the first position. Thus, the use of expert analysis in journal ranking requires primarily the exclusion of subjective evaluation factors and the selection of experts. Obviously the independent organizations, not affiliated with ranked journals, should make such ratings, to achieve the objectivity of final data.

Given the fact that the methods based solely on bibliometric information, or expert assessments, have certain drawbacks, experts try to combine both approaches to eliminate possible surges in the final rankings. Balatskii-Ekimova’s rating is an example of combination of assessment models used in the two previous ratings (2015) [2]. The list of 50 journals formed by 4 parameters is estimated by 5 quality parameters by 3 experts. It results in the list of journals put in order by the integral indicator. The choice of bibliometric indicators for the selection of publications is, in our opinion, a shortcoming of the proposed method. Hence, low-cited journals are included in the list. In particular, in the rating there are several editions, whose 5-year impact factor without self-citation is equal to 0.1 and below.

Thus, current ratings of scientific economic journals have certain drawbacks. Their elimination requires further research in the selection of criteria for ranking publications and improvement of their analytical methods.

The present study makes an attempt to classify journals of the RAS economic institutes by ranking the integral index, obtained by the method of multidimensional comparative analysis of bibliometric data.

Journal’s affiliation with the Russian Academy of Sciences is a kind of indicator of the publication quality. No wonder, experts believe that today the scientific success of a journal largely depends on its relationship with the RAS and the pursuit of academic traditions and standards, important in the formation of the leading journals of the country [2, pp. 111-112]. Scientists claim that the Academy’s primary mission to publish fundamental scientific results is projected on the activity of the RAS journals, playing an increasingly important role in the development of the Academy as a whole, the formation of key research priorities in the Russian space, the training of highly qualified personnel and the continuity of RAS generations [13, p. 182; 6, p. 172]. As RAS Corresponding Member G.B. Kleiner rightly stated, on the pages of these publications “first of all, fundamental knowledge should find a place – the result of fundamental academic research, and the fact of publication should be a step on the way from information as raw material for science to knowledge as its final socially recognized result” [6, p. 172]. According to experts, the academic journals “fulfil a certain patternforming function”, “scientific publications in such journals should serve as a model for authors and publishers by their structure, clarity of expression and preciseness of thought” [6, p. 173].

The above theses allow us to single out the RAS journals into a separate group of periodicals.

For the study we select all journals in economics published by the RAS economic institutions, subordinate to FANO. Before the RAS reform, these institutions were included in the Economy Section for the RAS Department of Social Sciences, i.e. were a single reference group. For the analysis we use publications only in economic and related sciences, not considering multidisciplinary journals and the ones, which have no indicators in the RSCI. Therefore, we analyze 18 publications.

The ranking of journals is based on the integral index obtained by rationing some bibliometric indicators (data of the Russian Scientific Citation Index). As the RSCI provides a number of different indicators, including minor, it is methodologically important to choose the indicators, which, first, ensure objective assessment of the publication’s importance in the academic circles and, second, are transparent and available for obtained results verification. At this stage we do not use expert judgment, because, as stated above, they introduce a certain percentage of subjectivity to final results. While not denying the importance of expert opinion, we believe that nowadays this issue requires certain methodological understanding in terms of rating Russian scientific journals.

In our opinion, citation metrics are key bibliometric indicators of scientific journal importance. It is reasonable to emphasize that the quantitative data on publication citation reflect the “impact of scientific publications on scientific community, its usefulness for other scientists” [5, p. 43]. We do not refer to the quality of publications of a journal. Nevertheless, when considering bibliometric indicators of periodicals, we can rely on citation metrics to prove that the work can be very significant with a high degree of probability.

The impact factor of a scientific journal is one of the basic scientometric indicators. It is introduced by E. Garfield and first used as a term in his article in the journal Science in 1955 [18, 1955]. Today the impact factor is the most common indicator of citation. C. Hoeffel, summarizing numerous opinions about feasibility of using impact factors, notes that, although the impact factor has not become an ideal indicator to measure the article quality, at the present time there is no other objective tool for scientific evaluation. Experience shows that in each specialty the best journals are the ones characterized by very strict requirements for the authors to get published and typically by a higher impact factor. Its wide spread as an indicator of quality and influence is caused, in Hoeffel’s opinion, by the fact that it correlates with the opinion, the scientists have about best journals in their disciplines [20, p. 1225].

Disclosing influence of articles, this index measures a number of links to one article included in a given set of publications for a certain fixed time period [17]. The time taken for publication of articles under analysis is called a “publication window”, and the time taken for publication of articles, links to which are considered in the impact factor calculation – a “citation window” [10, p. 82]. To determine the impact factor of a journal, a two-year publication window and a one-year citation window are traditionally used, i.e. the impact factor describes the average number of citations received per paper published in the journal during the two preceding years [10, p. 83]. The two-year period used is intended to make the statistic indicator relevant. According to Garfield, highly cited articles that reach the greatest “impact” are usually cited for months from the date of publication and certainly within a year after publication. Two years is enough to identify the so-called “hot papers” that have had the greatest impact in a particular field of scientific knowledge [16, 2005]. Hence, the use of a two-year citation window in the impact factor calculation is caused by the desire to identify periodicals publishing articles highly cited since publication and having the greatest impact in their field in the process of ranking journals relative to each other.

Experts believe that it is a two-year impact factor that is most often mentioned when comparing the level of journals nowadays [9, p. 131]. However, in some scientific disciplines, particularly in humanities and social sciences, the professional community can not accept new knowledge for a short two-year period of time. In this case, to calculate the impact factor it is advisable to use a wider publication window. So, in addition to a two-year impact factor researchers calculate a five-year impact factor, which characterizes the average number of links received per paper published in the journal during the five preceding years. Therefore, in fields, where the impact of published research should be considered for a long period of time, the five-year publication window helps measure the impact factor of journals more efficiently. Having the advantage over the two-year impact factor, such as an opportunity to align “surges” of certain articles with anomalous citation, according to experts, the five-year impact factor lacks responsiveness [19], as it lags behind in the reflection of new journals in the rating, whatever recognition the latter have won [3, p. 23]. Thus, in our view, when ranking journals by impact factor it is advisable to consider the values of both two-year and five-year indicators to neutralize their disadvantages and get a more objective result.

The impact factor value can be sufficient for ranking journals included in authoritative databases, comprised according to the strict criteria filtering out inferior publications. The data of the Science Citation Index are considered authoritative around the world. The Index is used as a tool to measure a level of scientific productivity due to a by-product – a database of the Journal Citation Report (JCR) and ranking of scientific journals by impact factor [19, pp. 63-64]. It is difficult to use this tool in relation to Russian scientific journals, particularly in economics, due to their weak representation in the international database Web of Science. Therefore, ranking of Russian economic journals is limited to the use of the Russian Scientific Citation Index data. As the RSCI does not select journals, the impact factor of a journal is not sufficient to rank scientific publications in the Russian realities. Experts note that the RSCI data are vulnerable to potential manipulation of impact factors by journal editorial boards. As a result, its high value does not guarantee the quality of a journal, as this value can be artificially inflated by self-and cross-citation of journals [8, p. 133]. Thus, when ranking Russian scientific journals by impact factor, the RSCI should apply additional quantitative characteristics of journals, in particular indicators, taking into account self-citations, and Herfindahl index values, calculated by citing journals and organizations of authors.

As noted above, citations received by the journal due to its own articles have a definite influence on the impact factor, as its standard definition includes self-citation. According to experts, such as R. Rousseau, the high level of self-citation indicates bad publicity of a journal [21, p. 425]. To neutralize the effect of self-citation on ranking of journals, it is necessary to consider the indicators characterizing self-citation, particularly a self-citation coefficient, which shows the proportion of links to the journal made by the journal itself of all the citations. In the RSCI the problem to separate self-citation is solved by calculating impact factors that take into account only links from other journals.

The Herfindahl index for citing journals is defined as the sum of the squares of the percentage shares of the number of articles citing this journal in relation to the total number of citations. The greater number of citing journals and the more even links to the journal are distributed, the smaller the value of this indicator. The Herfindahl index by organizations of authors is calculated as the sum of the squares of the percentage share of the number of articles published by different organizations in relation to the total number of articles in the journal in the current year, the organization is identified in. The more different organizations, the authors of which are published in the journal and the more even the publications are distributed between them, the smaller the value of this indicator. The maximum index value is equal to 10,000 and is achieved when all links are made from one journal or when a journal publishes authors from only one organization. Normalization in view of the Herfindahl index increases indices of the journals widely known in the scientific community, and decreases ratings of the journals with high self-citation or the ones using mutual citation to artificially improve their performance. According to A.A. Murav’ev, the Herfindahl indices “characterize the narrowness of authors’ circle and readership of a journal, acting as a specific measure of its “parochial mentality”. High values of these indices indicate that a journal is demanded and appreciated only by a small number of organizations and journals, which is hardly compatible with the status of a national edition” [8, pp. 137-138].

Thus, to overcome shortcomings of the impact factor as a traditional indicator of scientific importance of journals we use 4 bibliometric indicators to rank publications:

• 1.    RSCI two-year impact factor without self-citation (IF₂).

• 2.    RSCI five-year impact factor without self-citation (IF₅).

• 3.    Herfindahl index for citing journals (HI_J).

• 4.    Herfindahl index for organizations of authors (Ne_o).

The self-citation coefficient is not considered for normalization, since we use values of the impact factor without selfcitation. The matrix of initial indices is presented in table 1 . The journals in the list are arranged in the alphabetical order of their names. For each publication we specify the institute, it is affiliated with.

Table 1. Initial data matrix: bibliometric indicators of scientific journals of the RAS economic institutes in the RSCI (the NEL data as of December 1, 2015)

No.	Journal	RAS institute	IF 2	IF 5	HI J	HI о
1.	Vestnik of the Institute of Economics of the Russian Academy of Sciences	Institute of Economics of RAS	0.479	0.243	531	3390
2.	Economical Issues	Institute of Economics of RAS	1.957	1.508	125	1375
3.	Journal of the New Economic Association	A number of institutions, including Central Economic Mathematical Institute of RAS, Institute of Economics of RAS	0.746	0.707	251	1198
4.	Russian Journal of the Economic Theory	Institute of Economics, Ural Branch of RAS	0.574	0.440	703	414
5.	Population	Institute of Social and Economic Studies of Population of RAS	0.387	0.302	514	3309
6.	Applied Econometrics	Central Economic Mathematical Institute of RAS	0.429	0.625	810	3333
7.	Studies on Russian Economic Development	Institute of Economic Forecasting of RAS	1.341	1.308	187	1700
8.	Problems of Territory’s Development	Institute of Socio-Economic Development of Territories of RAS	0.533	0.348	1099	5652
9.	Spatial Economics	Economic Research Institute, Far Eastern Branch of RAS	0.886	0.688	490	3495
10.	Regional Agrosystems: Economics and Sociology (E-journal)	Institute of Agrarian Problems of RAS	0.067	0.064	1936	5838
11.	Regional Problems of Transforming the Economy	Institute for Social and Economic Research, Dagestan Scientific Center of RAS	0.264	0.234	3048	1071
12.	Region: Economics and Sociology	Institute of Economics and Industrial Engineering, Siberian Branch of RAS	0.432	0.385	276	2196
13.	North and Market	G.P. Luzin Institute of Economic Problems of Kola Scientific Centre of RAS	0.130	0.131	2402	3675
14.	ECO: All-Russian Economic Journal	Institute of Economics and Industrial Engineering, Siberian Branch of RAS	0.431	0.374	420	1318
15.	Economics and Mathematical Methods	Central Economic Mathematical Institute of RAS	0.483	0.670	319	1598
16.	Regional Economics	Institute of Economics, Ural Branch of RAS	0.800	0.548	230	879
17.	Economic Science of Modern Russia	Central Economic Mathematical Institute of RAS	1.036*	0.707	208	1243
18.	Economic and Social Changes: Facts, Trends, Forecast	Institute of Socio-Economic Development of Territories of RAS	0.915	0.592	545	2165
* As the index value for 2014 is not calculated in the RSCI, we consider the index for the previous year.

The classification of journals is based on the multidimensional comparative analysis method, which is widely used in economic research, in particular for comprehensive generalizing comparative assessment of enterprise performance [12, pp. 74-76]. The method of multidimensional comparative analysis is founded on the method of Euclidean distances helps take into account not only absolute index values of each journal, but also a degree of their proximity to the standard. For it the comparing indices of journals are expressed in fractions of the relevant standards, taken as a unit.

Thus, in the initial data table the maximum, taken as a unit, is defined for each index. The coefficients of other journals are calculated by dividing their initial indices by the maximum value of the standard. The obtained standardized coefficients are presented in table 2 .

The integral index for each journal is obtained by squaring standardized coefficients and adding up obtained results. The final ranking of journals includes ranking of integral indices ( tab. 3 ).

The obtained data allow us to single out the core consisting of 10 leading journals in

Table 2. Matrix of the standardized coefficients of scientific journals of the RAS economic institutes

No.	Journal	Index number
		1	2	3	4
1.	Vestnik of the Institute of Economics of the Russian Academy of Sciences	0.245	0.161	0.235	0.122
2.	Economical Issues	1.000	1.000	1.000	0.301
3.	Journal of the New Economic Association	0.381	0.469	0.498	0.346
4.	Russian Journal of the Economic Theory	0.293	0.292	0.178	1.000
5.	Population	0.198	0.200	0.243	0.125
6.	Applied Econometrics	0.219	0.414	0.154	0.124
7.	Studies on Russian Economic Development	0.685	0.867	0.668	0.244
8.	Problems of Territory’s Development	0.272	0.231	0.114	0.073
9.	Spatial Economics	0.453	0.456	0.255	0.118
10.	Regional Agrosystems: Economics and Sociology (E-journal)	0.034	0.042	0.065	0.071
11.	Regional Problems of Transforming the Economy	0.135	0.155	0.041	0.387
12.	Region: Economics and Sociology	0.221	0.255	0.453	0.189
13.	North and Market	0.066	0.087	0.052	0.113
14.	ECO: All-Russian Economic Journal	0.220	0.248	0.298	0.314
15.	Economics and Mathematical Methods	0.247	0.444	0.392	0.259
16.	Regional Economics	0.409	0.363	0.543	0.471
17.	Economic Science of Modern Russia	0.529	0.469	0.601	0.333
18.	Economic and Social Changes: Facts, Trends, Forecast	0.468	0.393	0.229	0.191

Table 3. Results of the comparative rating evaluation of scientific journals of the RAS economic institutes

Journal Index number Overall coefficient Integral index, R Rank 1 2 3 4 Economical Issues 1.000 1.000 1.000 0.091 3.091 1.758 1 Studies on Russian Economic Development 0.470 0.752 0.447 0.059 1.728 1.315 2 Russian Journal of the Economic Theory 0.086 0.085 0.032 1.000 1.203 1.097 3 Economic Science of Modern Russia 0.280 0.220 0.361 0.111 0.972 0.986 4 Regional Economics 0.167 0.132 0.295 0.222 0.816 0.904 5 Journal of the New Economic Association 0.145 0.220 0.248 0.119 0.733 0.856 6 Spatial Economics 0.205 0.208 0.065 0.014 0.492 0.702 7 Economics and Mathematical Methods 0.061 0.197 0.154 0.067 0.479 0.692 8 Economic and Social Changes: Facts, Trends, Forecast 0.219 0.154 0.053 0.037 0.462 0.680 9 Region: Economics and Sociology 0.049 0.065 0.205 0.036 0.355 0.595 10 ECO: All-Russian Economic Journal 0.049 0.062 0.089 0.099 0.297 0.545 11 Applied Econometrics 0.048 0.172 0.024 0.015 0.259 0.509 12 Regional Problems of Transforming the Economy 0.018 0.024 0.002 0.149 0.193 0.440 13 Vestnik of the Institute of Economics of the Russian Academy of Sciences 0.060 0.026 0.055 0.015 0.156 0.395 14 Population 0.039 0.040 0.059 0.016 0.154 0.392 15 Problems of Territory’s Development 0.074 0.053 0.013 0.005 0.146 0.382 16 North and Market 0.004 0.008 0.003 0.013 0.027 0.165 17 Regional Agrosystems: Economics and Sociology (E-journal) 0.001 0.002 0.004 0.005 0.012 0.110 18 economics and related disciplines of the RAS economic institutes (highlighted in table 2). These journals have significant citation metrics (impact factor) and a low self-citation level (values of the impact factor without selfcitation are considered). In addition, they can claim the status of all-Russian publications, as main citation to them is provided through links in a number of scientific publications and they are characterized by a fairly wide range of authors affiliated with different organizations.

It should be particularly noted that the final rating corresponds to the system of measurement parameters adopted in this study, i.e., the set of applied indicators and the method of their aggregation, and also depends on the quality of the RSCI data.

Table 4. Rating of scientific economic journals published by RAS institutes

Journal	R	Rank 1	IF 2 *	Rank 2	SI r	Rank 3
Economical Issues	1.758	1	2.068	1	5.261	1
Studies on Russian Economic Development	1.315	2	1.450	2	3.211	2
Russian Journal of the Economic Theory	1.097	3	0.830	7	0.342	10
Economic Science of Modern Russia	0.986	4	1.036**	4	1.522	3
Regional Economics	0.904	5	0.933	6	1.103	5
Journal of the New Economic Association	0.856	6	0.781	8	1.231	4
Spatial Economics	0.702	7	0.986	5	0.651	7
Economics and Mathematical Methods	0.692	8	0.540	13	0.896	6
Economic and Social Changes: Facts, Trends, Forecast	0.680	9	1.090	3	0.556	9
Region: Economics and Sociology	0.595	10	0.446	15-16	0.594	8
ECO: All-Russian Economic Journal	0.545	11	0.560	12	0.294	13
Applied Econometrics	0.509	12	0.446	15-16	0.307	11
Regional Problems of Transforming the Economy	0.440	13	0.579	10	0.069	16
Vestnik of the Institute of Economics of the Russian Academy of Sciences	0.395	14	0.577	11	0.219	14
Population	0.392	15	0.509	14	0.297	12
Problems of Territory’s Development	0.382	16	0.715	9	0.175	15
North and Market	0.165	17	0.320	17	0.044	17
Regional Agrosystems: Economics and Sociology (E-journal)	0.110	18	0.152	18	0.023	18
* Values of the two-year impact factor (including self-citations) are presented. ** As the index value for 2014 is not calculated in the RSCI, we consider the index for the previous year.

The drawback of the proposed method to rank journals consists in the impact of high values of certain indices on the end result. So, “Russian Journal of the Economic Theory” ranks the 3d only due to the highest coefficient of the latter parameter in the reference group, i.e. the best values of the Herfindahl index by organizations of authors. At the same time, other values, characterizing citation, are average. Although it is a single situation, we can see that the proposed method to rank journals can be improved in the future.

As the international practice has no recognized methods to validate the rating data, we compare the received data with the lists of economic journals, based on the RSCI bibliometric indicators (Murav’ev’s rating [7, 8] and Balatskii-Ekimova’s rating [2]) to verify the final results.

Of the first ten journals of our list 4 publications are presented in Murav’ev’s rating (2013) in the A category. This is “Economical Issues”, “Journal of the New Economic Association”, “Economics and Mathematical Methods”, and “Economic Science of Modern Russia”. This category includes the journal “Applied Econometrics” (in our list –position 12). Two other journals “Spatial Economics” and “Studies on Russian Economic Development” belong to the B category, which comprises 14 publications, also having rather high private ranks. The journals “Economic and Social Changes: Facts, Trends, Forecast” and “Regional Economics” find themselves in the C category (includes 24 books). Three categories include

55 journals, which the author characterizes as the most influential Russian publications in economics and related disciplines. The journals of the A and B categories are defined as the most significant publications and the ones of the A category – as the leading national economic journals.

In Balatskii-Ekimova’s ranking (2015) 7 publications from our list belong to the group of the best national journals in economics, called as a “Diamond” list of journals (DLJ) by analogy with foreign tradition. This is “Economical Issues”, “Economics and Mathematical Methods”, “Applied Econometrics”, “Journal of the New Economic Association”, “Studies on Russian Economic Development”, “Spatial Economics”, and “Economic Science of Modern Russia”. E.V. Balatskii notes that of 13 DLJ publications a half is anyhow connected with the Russian Academy of Sciences. In his opinion, it proves an important role of academic standards and traditions in the formation of leading journals of the country [2, p. 112].

Given the substantial number of concurrences with other lists, we can conclude that the presented ranking does not contradict previous research fundamentally. Therefore, the proposed composition of criteria and method of their aggregation can be used to rank scientific journals.

We should mention weak correlation of our integral rating and the RSCI journal ranking by impact factor and the Science Index rating, calculated in the RSCI by the special method, taking into account several criteria ( tab. 4 ).

As judged by rank comparison, the ranks matching in all classifications are observed only among the journals with rather high indices. Thus, our results also show that the RSCI rating can not be used as an ultimate criterion for ranking scientific journals.

To sum it up, we should note that the acquired results help address the question to select criteria, in particular bibliometric, for assessing scientific journals. The proposed composition of criteria and method of their aggregation is quite suitable for ranking scientific journals, which is supported by sufficient correlation with other rankings. The shortcomings of the proposed method revealed at this stage determine the prospects of research in the clarification of these criteria and their aggregation.

In general, we can say that the integral values, obtained by the method of multidimensional comparative analysis of several significant bibliometric indicators of the publications in the RSCI, has allowed us to rank them and identify the core of ten leading journals in economics and related disciplines of the RAS economic institutes. The results of the comparative rating evaluation of scientific journals can be considered by these institutions, when working out strategies to promote their publications. The improved performance by a number of positions is directly associated with activities of an editorial board, whose efforts can be directed on increasing the requirements to quality of publications and their peer review, monitoring self-citation and expanding the geography of authors.