The impact rating of academic journals in economics: ranking criteria and methodology

The quality of scientific journals and the problem of selecting those with the greatest impact are the issues that are gaining importance due to the rapidly increasing number of publications ¹ . These issues came to the fore in January 2018, when the President of the Russian Academy of Sciences Aleksandr Sergeev said at a press conference that state assignments for academic institutions for 2018 will be extended; but when determining the volume of additional publications, not so much their number as the quality and impact of the papers will be welcomed ² . In the subsequent recommendations on the adjustment of research plans and state assignments for 2018, it is proposed to take into account the requirement for quartiles of journals with regard to additional publications. Thus, we can say that the priorities of publication policy for scientific organizations are adjusted in favor of quality indicators.

Scientists are urged to publish their papers in high-rated journals. And if in the case of foreign publications it is clear that we are talking about those that are indexed in international scientometric databases and are ranked in them by quartiles according to the impact factor, then the situation with domestic publications is less clear. Obviously, the results of all the studies of Russian scientists cannot be published in foreign journals alone. In order to ensure the competitiveness of our country in the world, it is necessary to form the core of Russian scientific journals that can, along with the leaders of international scientific periodicals, influence the development of all scientific areas.

So far, there are no official lists of leading publications broken down by field of science. Unfortunately, Russia does not have a national database indexing leading scientific journals. It is clear that the Russian Science Citation Index (RSCI) does not solve this problem, because it includes journals on the declarative principle and does not set strict criteria for them so as to be able to select only high-quality publications. Thus, the task of evaluating scientific journals and finding the criteria for identifying those with the highest impact now rests with the academia. Experts try to find a solution by ranking the journals on various parameters. The recent ratings of economic journals have been widely discussed in the scientific community. Not only the criteria selected for evaluating publications in different ratings, but also the very idea of ranking the journals and the ability to approach the assessment of their quality have been subjected to critical reflection. The question of what should be the basis of ranking – bibliometric indicators or expert assessments – was a matter of heated debate.

The previous issue of the journal Economic and Social Changes: Facts, Trends, Forecast (Discussion Platform section) contains an article by E.V. Balatsky and N.A. Ekimova entitles “Opportunities for the consolidation of rating products in the Internet environment”. Describing the experience of implementing a project on consolidation of rating products on a single information and analytical portal, the authors came to the conclusion that “there is currently a certain confrontation between the rating movement and the expert community, which extends to the confrontation between quantitative and qualitative ratings” [1]. The experts note the weaknesses of the radical manifestation of both approaches and believe that the mutual adjustment of these areas will continue for a long time, but at the same time “the growing practice of making ratings and the dialogue of rankers with the wider expert community will contribute to improving the quality of both the former and the latter” [1].

We certainly agree that expert evaluation has its strong points; still we share the view that quantitative factors may have an advantage in the methodology of assessing journal relevance and impact [2]. The use of quantitative indicators enables us to compare a large sample of publications, which is extremely difficult to do with the use of expert analysis. Moreover, despite the fact that quantitative indicators evaluate formal rather than substantive aspects of the journal, there is a list of formal requirements, the compliance with which is considered mandatory for a high-quality journal.

Earlier we have already made an attempt to make a rating of academic economic journals of RAS institutes based on the analysis of bibliometric indicators [3]. We got the results suitable for ranking and quite comparable with the data of other ratings. Nevertheless, the top list of journals contained the so-called outlying cases, i.e. the publications that were in the top part of the rating only on the basis of one indicator that turned out to be the best in the reference group.

While continuing the study of individual ranking criteria, we came to the conclusion that in order to assess the impact of scientific journals it is necessary to take into account such an important indicator as the number of highly cited articles. By this we determine not only the impact of the journals, but also identify those that publish breakthrough research findings among them.

Thus, the goal of our present work is to establish and substantiate the criteria for identifying high impact journals and use them for ranking. The results of the research are presented in the form of a rating of economic journals affiliated with organizations of the Russian Academy of Sciences.

In order to provide the conditions for proper use and correct interpretation of the ranking results, we have named the resulting list the impact ³ rating , which we define as a type of rating of academic journals that is based on the analysis of bibliometric indicators and that reflecting the impact of the academic journals it contains. We understand the impact of the journal as its influence on the scientific area, its ability to accumulate the results of breakthrough research, and the notion of how scientists themselves perceive the scientific authority and prestige of the publication.

It is necessary to explain that when comparing journals on bibliometric parameters we do not insist on the dominance of such an assessment, but admit that it often requires expert opinion to be considered along with the results obtained. But in this case, first of all there is a question of selecting the experts and excluding subjective factors from the assessment. Since the economists themselves note that the modern Russian expert community experiences a decline in the academic ethics index and is unable to evaluate its colleagues objectively [1], we believe that in order to ensure transparency of the final data, the preparation of expert ratings should be carried out by independent organizations that are not affiliated with the journals undergoing evaluation.

The results that we have obtained can be used not only to identify journals that have a high impact in their field of science and publish influential articles. They can also be taken into account by scientific organizations for determining the strategic priorities of their publication activities. Editors can adjust the development programs of their journals, focusing on indicators that can be improved by increasing the requirements for the level of publications and the quality of their review, making the requirements for self-citation more strict, expanding the geography of authors, etc.

Theoretical basis for ranking academic journals

Due to the increase in the number of published scientific journals, the issue of their differentiation and choosing the leading ones among them comes to the fore.

In foreign practice, bibliometric traditions in ranking economic journals are quite strong [4], although one of the earliest ratings developed by R. Hawkins, L. Ritter and I. Walter was based on the analysis of expert opinions and did not take into account quantitative data [5]. The scientometric approach was developing alongside the expert opinion approach and was associated with the emergence of journal ratings developed on the basis of citation analysis [6; 7]. And if earlier ratings were based on the use of simple methods, for example, on the calculation of the ratio of citations to the number of printed characters published for a certain period [8], then later researchers attempted to introduce more complex methods of citation analysis in order to correct methodological limitations of the impact factor, which was used as the main bibliometric parameter [4; 9]. Modern research has also introduced new approaches to measuring the potential impact of journals not only in the academia, but also in the wider community. For example, the core of economic journals was formed on the basis of their citation in major textbooks on economics [10]. We should point out that foreign scientists, when using citation analysis data in the ranking process, consider citations as an indicator of quality, which reflects at least the impact [11] and apply these data to obtain the quality index for academic journals [6]. Speaking about the purpose of citation analysis techniques in general, foreign experts believe that they can be used primarily to assess the importance and impact of individual journals and their role and position in the system of scientific communication; in addition, they can help understand how scientists themselves perceive the quality and impact of publications [12].

The first attempts to identify the leading economic journals in Russia that were made in the early 2000s were based on expert opinions. In this way, the composition of the list of top journals in economics was determined, a comparative analysis of which is presented in an article by S. Aukutsionek and G. Churkina, published in 2002 in the journal Voprosy Ekonomiki [13]. It should be noted that in the publications of that time that used the information about the leading scientific journals, the experts did not focus on the criteria and procedure of their selection. For example, I.G. Dezhina and V.V Dashkeev mentioned only the fact that the list of 12 economic journals, which was used by the authors as a source for identifying leading economists, was determined by expert assessments from among the most famous ones [14].

To date, several methodological approaches have been formed to identify the impact of academic journals (Tab. 1) : 1) bibliometric approach, based on the analysis of scientomet-ric indicators (Murav’ev’s rating, 2013 [15]; Tret’yakova’s rating, 2015 [3]); 2) expert approach, built on the sociological assessments of opinions of the scientific community (project of the National Research University Higher School of Economics, 2015 [16]; Rubinshtein’s rating, 2017 [17]); 3) expert and bibliometric approach that combines bibliometric analysis with expert assessments (Balatsy–Ekimova’s rating, 2013 [18]); 4) network approach , which identifies system-wide important scientific journals in the networks arising from cross-citation (Aleskerov et al., 2016 [19]);

Table 1. Main methodological approaches to ranking Russian economic journals

Methodological approach	Rating product	Issue date	Developer	Organization
Bibliometric	Rating of journals in economics and allied disciplines	2013	A.A. Murav’ev	Saint Petersburg University; NRU HSE (Moscow)
	Rating of academic journals of RAS economic institutes	2015	O.V. Tret’yakova	Vologda Research Center of RAS
Expert	HSE rating of Russian academic journals (economics)	2015	Office for Research Evaluation at the National Research University Higher School of Economics	NRU HSE (Moscow)
	“Cluster” rating of Russian economic journals	2017	Rubinshtein et al.	RAS Institute of Economics; NRU HSE (Moscow)
Expert and bibliometric	Rating of leading Russian economic journals	2013-2016	E.V. Balatsky, N.A. Ekimova	Financial University under the Government of the Russian Federation
Network	Rating of economic journals based on cross-citation analysis	2016	F.T. Aleskerov et al.	NRU HSE (Moscow)
Aggregation	Aggregated rating of scientific journals in economics and management	2016	A.N. Subochev	NRU HSE (Moscow)
	Consensus rating of leading Russian economic journals	2017	E.V. Balatsky, N.A. Ekimova	Financial University under the Government of the Russian Federation

• 5)    aggregation of existing rating products (Subochev’s rating, 2016 [20]; Balatsky– Ekimova’s consensus rating, 2017 [21]).

According to the scientific and expert community, none of these approaches is flawless. Weak points of the ratings are said to be as follows: bibliometric indicators are often selected arbitrarily and they can have a weak correlation with academic authority of the journals; besides, the procedure for aggregating the indicators or expert assessments can be insufficiently substantiated; in addition, the surveys of experts can lack representativeness [22]. Experts believe that the existence of different ratings of Russian journals that are based on the same indicators and provide different ranking results is an evidence of unreliability of the ratings [23].

The intensified rating movement aimed to evaluate leading Russian economic journals has urged the academia to discuss methodological approaches to ranking, the appropriateness of using bibliometric indicators in these approaches, and the comparability of the results obtained. There are different opinions on various rating products, up to completely opposite viewpoints. In 2016, the website of the journal Neergodicheskaya ekonomika [Nonergodic Economics] opened a special information and analytical portal “Ratings”; in the current situation, this can be considered an attempt to consolidate rating products and establish a dialogue between rankers and the expert community in order to smooth the contradictions between quantitative and qualitative assessments and develop the correct attitude toward ratings, which would help eliminate the errors related to these assessment tools [1].

The impact rating that we have developed reflects the degree of impact of scientific journals and is based on the analysis of citation indicators. The traditional idea of citation as an indicator of impact and an instrument for assessing scientific contribution follows from the theoretical works of R. Merton, who believed that if the work of a scientist remains unnoticed and is not used by other members of the academia, then the value of such work is doubtful [24]. Approaching the interpretation of the role of citation from different aspects, researchers note that it is somehow an indicator of “the usefulness and importance of the work” (Garfield [25]), “the authority of the cited work”, since authors usually cite authoritative articles and avoid “trivial” and “irrelevant” ones (Gilbert [26]). Authority here is understood as “the potential impact of the publication on the activities carried out around the research”; influence is defined as real impact (Martin, Irvine, [27]). When asked what exactly makes highly cited works important and authoritative, some researchers talk about the fact that “the peers recognize the cognitive value of the sources that have become influential because they are highly cited” and characterize citation as a “criterion of intellectual impact” (Zuckerman [28]).

We share the opinion of experts and agree that citation can be used as a tool to measure the impact that the work has on the community as a whole (S. Cole, J.R. Cole [29]; H. Moed, [30]). Taking citation rate as a measure of impact of publications, we have analyzed economic journals affiliated with organizations of the Russian Academy of Sciences, and made an impact rating of these journals.

Ranking methodology

We use the Russian Science Citation Index as a tool to assess the impact and authority of academic journals in economics. We apply the multidimensional comparative analysis technique based on the distance method; the technique is widely used in economic research to carry out a comprehensive comparative evaluation of the economic performance of enterprises. In relation to scientific journals, this method enables us to consider not only the absolute values of bibliometric indicators of each journal, but also the degree of their deviation from the standard.

In order to make an impact rating that would show the relevance and authority of academic journals we adjust the composition of the criteria. We use the following four indicators:

• 1.    RSCI two-year impact factor without self-citation (IF ₂ ).

• 2.    RSCI five-year impact factor without self-citation (IF ₅ ).

• 3.    Herfindahl index for the citing journals (HI _J ).

• 4.    Number of highly cited papers (HP).

In our opinion, these indicators, on the one hand, allow us to assess with a high degree of objectivity the level of relevance and authority of the publication; on the other hand, which is no less important, the above indicators are transparent and accessible if there is a need to verify the results.

The impact factor devised by the American scientist Eugene Garfield as a tool for measuring the value of journals by calculating the average number of citations per article for a certain period of time [31] is traditionally used as an indicator reflecting the scientific prestige [32] and authority of a journal [34] and its impact on the relevant scientific field [35]. According to Hoeffel, widespread use of the impact factor as an indicator of journals’ relevance and authority is due to the fact that it correlates very well with the opinion that has developed among scientists about the best journals in their disciplines [36].

We use two indicators of the impact factor (two-year and five-year) as the criteria for ranking journals because first, we want to identify the journals that publish the articles that have the greatest number of citations since their publication and the greatest impact in their field; and second, because it is necessary to smooth out the outlying cases of individual articles with abnormal citation rate by taking into account the impact of the papers for a longer period of time, which allows us not to understate the rating of the journals the papers in which receive a considerable if slower response of the academia. In order to neutralize the effect of self-citation, the high level of which, as R. Rousseau points out, shows that the journal is not widely-known [37], we use impact factor values that take into account only the links from other journals. If a journal has an English version, then we take the values of the impact factor taking into consideration the translated version without self-citation.

When ranking the journals we want to take into account not only the rate of citation, but also its scale, i.e. our goal is to identify the journals well-known in the scientific community and, simultaneously, to lower the rating of the journals that are cited by a narrow circle of other publications and that use mutual citation in order to improve their indicators artificially. To achieve all this we apply normalization taking into account the Herfindahl index for the citing journals. Its value is defined as the sum of the squares of the percentages of the journals citing this one in the total number of citations. The greater the number of the citing journals and the more evenly the references to the journal under consideration are distributed among them, the smaller the value of this indicator. The maximum value of 1,000 is reached when all references are made from a single journal. High values of this index show that the journal is in demand and is valued highly only by a small number of journals, and according to experts, such a situation is incompatible with the nationwide status of the publication [15].

In our opinion, when identifying the most in-demand scientific journals in different thematic areas, the information on the number of highly cited articles published in them should be taken into account. We believe that in addition to the impact this indicator will determine the scientific level of the journal and its ability to accumulate breakthrough research findings. The values of the two-year and five-year impact factors, as well as the Herfindahl index for the citing journals were established by the values already calculated in the RSCI. As for the number of highly cited articles, we calculated it on our own.

In order to determine highly cited publications we used basic principles of the methodology of the international database Essential Science Indicators ⁴ ; the database considers a paper as highly cited if it is among 1% of the world’s most cited papers among those published in the same year and in the same scientific field. Experts traditionally consider such articles as being of the highest quality in terms of international recognition of scientific findings of researchers from a certain country (Aksnes, Sivertsen [38]; Garfield [39]; Glanzel, Schubert, [40]; Tijssen, Visser, Leeuwen [41]; Kotsemir [42]), emphasizing their relevance for the science of the top level (Pislyakov [43]).

We believe that the main principles of the methodology for determining highly cited publications can be applied not only to Web of Science data, but also to citation indicators obtained from other analytical systems. For example, to evaluate research performance of Russian economists whose works are not represented widely in international databases, it is possible to get more complete data on their publications from the Russian Science Citation

Index. Of course, in this case we are not talking about international recognition of the results reflected in the publications and about their impact in the world science as a whole, but we can assess their importance for the Russian segment.

To determine the number of highly cited publications, it is necessary to compare the papers in the journal under consideration and in other journals. Thus, the choice of such publications will depend not only on the quality of the journal’s own papers and their citation, but also on its standing in comparison with other journals on the same subject. Since it is incorrect to compare the works of different years due to the fact that some of them could get more citations because of their “age”, the comparison is carried out in a subset of publications issued in the same year.

We analyzed a large sample of articles published in 2016 in journals indexed in the RSCI on the subject “Economics. Economic sciences”. All publications were arranged in descending order of their citations, after which we determined the upper section of 1% of their total number. Using the number of citations of the last article included in the upper section we set a threshold number of citations that a publication must receive in order to become highly cited. Then for each journal from the reference group we formed a list of articles whose number of citations meets the threshold. After that, we evaluated the quality of each article. For our purpose, we have excluded self-citation and those citations that we conventionally call “cluster” citations. We are talking about the cases when one article is cited many times by different authors in a single collection or journal. Obviously such citation does not indicate the availability of breakthrough research findings and cannot be taken into account in identifying the most relevant publications. Having “cleared” the works of self-citation and “cluster” citations, we established for each journal the number of publications, the number of citations of which remained not lower than the threshold, and defined them as highly cited for the journals of this reference group.

As mentioned above, the overall ranking of the journals was carried out with the help of multidimensional comparative analysis based on the distance method. In each column of the source data matrix we defined the maximum element (max a_i ), which was taken as a unit. The matrix of standardized coefficients ( x_ij ) was created from the values obtained by dividing each input factor in the column ( a_ij ) by the maximum (optimal) element of the model journal (max a_i ):

aij xii = j    max ai

.

We carried out “reverse” normalization for the indicator reflecting the Herfindahl index for the citing journals; the best index value is considered the one that is the lowest.

At the next stage all elements of the matrix of standardized coefficients were squared. From the sum of the squares of the indicators selected for journal evaluation, we extracted the square root to obtain an integral index of the generalizing rating score ( R_j ). The algorithm of calculation is as follows:

Rj = V^2 + x^2 + xi32 + xi42 .

The final rating of the journals is based on ranking the integral indicators ( R_j ), the position of each journal is determined by the level of its impact and importance for the scientific community: the first place is occupied by the journal with the highest value of the integral index; the second place – by the journal with the second best result, etc.

Results of ranking academic journals in economics according to bibliometric parameters

We have analyzed bibliometric indicators of economic journals that are affiliated with the organizations within the academic sector of science that were included in the Economics Section of the Social Sciences Department of the Russian Academy of Sciences before the reform of RAS and represented a single reference group; we have ranked these journals by analyzing the integral indicator obtained with the help of multidimensional comparative analysis of bibliometric data. For the purposes of our analysis we have selected 16 journals that are included in the Russian Science Citation Index in the subject area “Economics and Economic sciences”. We think that the results of our analysis can be useful for other scientific publications who are interested in their own development and in entering international information systems. In addition, the results we have obtained can be taken into account by the organizations that publish these journals, so that they could work out strategies to develop the publications. The initial data and the results of journal ranking are presented in Table 2 .

Most of these journals enjoy considerable authority in the Russian scientific community, as indirectly evidenced by the distribution of impact factors.

In the reference group under consideration, we compared the values of the two-year and five-year impact factors of RSCI without selfcitation for 2016 with their median values in the group of journals in the subject area “Economics. Economic sciences” (data of the RSCI for April 2018). We found out that 410 publications have the value of the two-year impact factor greater than 0. The median of the distribution of two-year impact factors is 0.293. This means that half of the journals have an impact factor above 0.293, and half – below this value. The median of the values of five-year impact factors was 0.260.

The impact factors of 15 journals in the analyzed reference group significantly exceed the median indicator; this fact shows a sufficiently high level of their citation, and consequently, the demand in the scientific community and the importance for the scientific field (Fig. 1) .

As for the Herfindahl index for the citing journals, its values are low (less than one thousand) for 14 journals. Consequently, they are generally quite well known in the scientific community.

Let us describe in more detail the calculation results for the indicator of the number of highly cited papers, which is absent in the RSCI and which we performed on our own. The analysis was carried out on a large array of economic journals for 2016. The sample consisted of 606 journals with 52,220 articles. With the use of the methodology of the international database Essential Science Indicators we distinguished 547 highly cited articles published in Russian economic journals in 2016 (according to RSCI data as of January 2018).

The threshold value of the number of citations that a work published within the specified year had to receive in order to enter the top 1% of the highly cited works is 11. The total number of references to the publications included in the upper section of the most cited papers has exceeded 10 thousand. Thus, the share of 1% of all publications in Russian economic journals in the RSCI accounts for 8% of the total citations, i.e. one citation in twelve belongs to a highly cited paper.

The data on the number of highly cited articles in economic journals of the academic sector are presented in Figure 2 .

Having analyzed the relative indicators showing the proportion of articles that are published in the journal and that can be considered highly cited, we can say that the journal Voprosy ekonomiki has greatest share of highly cited

Figure 1. Distribution of economic journals published by RAS institutes, by the value of the two-year impact factor RSCI without self-citation for 2016 (data of the RSCI as of April 2018)

ф 7.288

• 4.650

2 . 538 ₂. 5

2 . 5

1 . 5

0 . 5

Two-year impact factor RSCI Five-year impact factor RSCI

Median of distribution of two-year impact factors Median of distribution of five-year impact factors

The numbers in the graph denote the following journals:

• 1    – Voprosy ekonomiki [Economic Issues]

• 2    – Problemy prognozirovaniya [Studies on Russian Economic Development]

• 3    – Ekonomika regiona [Economy of Region]

• 4    – Prostranstvennaya ekonomika [Spatial Economics]

• 5    – Ekonomicheskie i sotsial’nye peremeny: fakty, tendentsii, prognoz [Economic and Social Changes: Facts, Trends, Forecast]

• 6    – Zhurnal novoi ekonomicheskoi assotsiatsii [Journal of the New Economics Association]

• 7    – Problemy razvitiya territorii [Problems of Territory’s Development]

• 8    – Prikladnaya ekonometrika [Applied Econometrics]

• 9    – EKO [ECO Journal]

• 10    – Region: ekonomika i sotsiologiya [Region: Economics and Sociology]

• 11    – Vestnik Instituta ekonomiki Rossiiskoi akademii nauk [Bulletin of the Institute of Economics of the Russian Academy of Sciences]

• 12    – Regional’nye agrosistemy: ekonomika i sotsiologiya [Regional Agricultural Systems: Economics and Sociology]

• 13    – Ekonomika i matematicheskie metody [Economics and

Mathematical Methods]

• 14    – Ekonomicheskaya nauka sovremennoi Rossii [Economics of Contemporary Russia]

• 15    – Zhurnal ekonomicheskoi teorii [Russian Journal of Economic Theory]

• 16    – Regional’nye problemy preobrazovaniya ekonomiki [Regional Problems of Transforming the Economy]

publications in the total number of the articles. We can conclude that in 2016, every third publication of this journal was among the highly cited ones.

Having ranked the journals on the basis of the integral index obtained by the method of multidimensional comparative analysis of bibliometric indicators of the journals according to the stated parameters, we identify the core of top ten economic journals affiliated with scientific organizations of the Russian Academy of Sciences (Tab. 2). Judging by the results, the top of the rating includes publications that are well-known in the scientific community and that have an impact on the development of economic science in the country. This can be concluded from their high citation rates that indirectly show the scientific prestige of the journals. Their widespread popularity in Russia is confirmed by the considerable number of the citing journals, as evidenced by the low values of their Herfindahl index. All this allows us to

Figure 2. Number of highly cited articles in economic journals of the academic sector, 2016 (RSCI data as of January 2018)

say that the publications included in the core of the list are among the top Russian journals in economics and to determine their status as nationwide.

We characterize the level of impact of the journals included in the top five as high. Today, their authority is recognized not only by Russian economists, but also by the international scientific community, since they are included in the main international scientometric databases: Voprosy ekonomiki (WoS: ESCI, Scopus), Ekonomika regiona (WoS: ESCI, Scopus), Zhurnal Novoi ekonomicheskoi assotsiatsii (WoS: ESCI, Scopus), Problemy prognozirovaniya (Scopus), Ekonomicheskie i sotsial’nye peremeny: fakty, tendentsii, prognoz (WoS: ESCI). As we can note, these databases traditionally select the most influential journals from different scientific areas, which promote advanced ideas and meet high standards of the quality of scientific content. Prior to being accepted by these databases, the journals undergo an evaluation procedure carried out by the expert group on formal and qualitative criteria. That is, we can say that the scientific authority of the publications under consideration is confirmed not only by bibliometric indicators, but also by independent expert opinion.

In order to verify the final results we compare them with the results of other ratings for the similar time period (Tab. 3) . First of all, we are interested in the Rating of Russia’s leading economic journals–2016 developed by Evgenii V. Balatsky, Director of the Center for Macroeconomic Research at the Financial University under the Government of the Russian Federation, and his colleague Natal’ya A. Ekimova ⁵ . Having compared the two lists we see that Balatsky–Ekimova’s diamond list contains nine journals out of the ten that form the core of our rating. At the same time, the positions of the top five publications from our list coincide with the order of the journals affiliated with RAS organizations in Balatsky’s rating.

Table 2. Results of comparative rating of economic scientific journals published by organizations of the academic sector

ш о го Е о О

о О

сч

CO

^г

ш

со

1^-

ОО

СП

р

^Z

см

co

p

in

о го

g

tn

2

го

ОО со in

сч in in

8

in ОО

5

2

2

in co

2

Го

2

Е со

g

сч in

in in

ГО

8

m

in S

in сч

CD

Го

ro

CD

CD

m

to" О го

X*

g

ю

g

ОО

in

oo

ОО

8

g

g

g

g

g

g

g

g

х~

g

ОО

in in

2

2

ю сч

2

CD

CD

CD

CD

ro

in

8

х~

g

р

8

in сч

со

8

р

in ОО

g

s

8

8

s

in

со

g

х"

g

го

8

CD

сп

8

р

in

P

2

in

ro

8

CD

§

8

х го Е о "ш со

><г to О О

X

g

сч

g

8

CD

8

ст>

Го

g

g

g

g

g

g

g

g

X

g

in in in

ro

8

Го

СЧ in

ю

Го

СЧ in

Го

ro

2

ro

g

2

ro

X

g

OD

oo

СЧ in

2

CD

СП

8

СЧ

Го

ro

ro

co in

CD

2

§

х"

g

2

8

oo

Го

in сч

2

in

Го

CD

3

CD

ro

8

X го Е

to О го

□_ х

CO

^

CD

CO

CO

СО

CD

CD

CD

CD

CD

CD

CD

CD

X

co

ГО

P

ro

ГО

in in

Го

8

S

8

8

CM

О

P

ll

Ю co

oo

oo

8

СП

8

Го

ю

8

Ю co

OD in

ro

S UD

UD

8 СЧ

Ll"

ro

8 СЧ

oo

oo in СЧ

ГО

CO

g

ro

in o>

8 co

СЧ

m in

in in co

2

о

Е Z

Ё о о "ш to* О p

ro о О) Ё о о Ш

о "to E о о cd о с Ё с Ё 5 8 IXJ го

ro о ГЧ о О) о Е ср о

ГЧ о О) о i 1 to (D $ s Е о с Е о Е ш о.

о "to to О X Ё о о "о

о ^ ш

Ё о о "о to о £

E о о "ш

о о Е CD О to Е О : = с: to S О Ш ОТ

O) О О 'to о to Ё о о CD О О) X

о CD E E Ё о о Ш

О E CD О £

о CD О "to E о о "ш

Ё о о "о E "to о О) -C О) га O 2 5

Ё >4 ° >o о ” s о O) c> X CL

6 X

сч

СО

in

со

г--

ОО

СП

о

^z

сч

co

р

in

ro

Table 3. Comparison of the results of the impact rating of the journals from the academic sector with the data of other ratings of Russian economic journals

Journal	Impact rating, 2016	Balatsky–Ekimova’s rating, 2016 Rank / sequence order of RAS journals	Balatsky–Ekimova’s consensus rating, 2017 Grading / rank	Rubinshtein’s rating, 2017 Category / rank
Voprosy ekonomiki	1	1 / 1	А / 2	A1 / 2
Ekonomika regiona	2	3 / 2	D / 27	-
Zhurnal novoi ekonomicheskoi assotsiatsii	3	5 / 3	А / 3	A1 / 1
Problemy prognozirovaniya	4	6 / 4	В / 10	A3 / 8
Ekonomicheskie i sotsial’nye peremeny: fakty, tendentsii, prognoz	5	10 / 5	E / 46	-
Vestnik Instituta ekonomiki Rossiiskoi akademii nauk	6	-	D / 21	B1 / 14
EKO	7	39 / 9	D / 19	-
Prostranstvennaya ekonomika	8	20 / 7	B / 9	A3 / 11
Prikladnaya ekonometrika	9	12 / 6	А / 5	A2 / 4
Ekonomicheskaya nauka sovremennoi Rossii	10	21 / 8	B / 11	B1 / 15

A weaker degree of correlation can be seen when comparing our ranking results with the data of the Consensus rating of leading Russian economic journals [21]. Although it contains all the ten journals that form the core of our rating, their position relative to each other is significantly different. We believe that this is due to different ranking criteria and the time gap between our list and the individual rating products, the results of which are summarized in the Consensus rating.

The lowest correlation is observed when we compare the results of our rating with the rating developed under the guidance of A. Rubinshtein and which takes into account only the data of expert surveys [17]. In our opinion, this is due not so much to the lack of significant links between bibliometric indicators and the opinion of the expert community based on intuitive views of economists about the scientific authority of journals, as to the composition of the initial sample of the journals. According to the rating developers, the initial list of analyzed journals has been formed on the basis of the list of publications included in the RSCI, the principles of construction of which have aroused many questions from the expert community, which were discussed on the pages of scientific publications [44]. Taking into account the known shortcomings of the list, the authors have made an attempt to adjust the selection of publications. Nevertheless, we see that it does not include even those journals that are included in global citation indexes and whose quality and level of impact are confirmed by independent expert opinions at the international level. We believe that if the sample and, possibly, the geography of the experts were expanded, the results of the rating based on the data of sociological surveys would be different, and the degree of their correlation with the results of our rating would be higher.

Judging by a high degree of correlation between our rating and Balatsky–Ekimova’s rating that is based on a combination of bibliometric and expert assessments, we can assume that the composition of the criteria we have chosen and the method of their aggregation we have used may be suitable for ranking not only publications of the academic sector, but also all Russian economic journals. Some difficulties associated with the use of this method to rank a wide range of journals are seen in the procedure for calculating the number of highly cited publications, in particular, the need for additional qualitative evaluation of citations that each such article received.

Conclusion

Summing up, we should note that the results we have obtained allow us to approach the question of the choice of criteria, in particular bibliometric ones, to evaluate academic journals. The composition of criteria we propose has been adjusted in comparison with the sets of criteria used in the previous rating [3]. Having introduced the indicator of the number of highly cited publications we were able to deepen the analysis of citation and to rely on the characteristic of its ability to accumulate “breakthrough” articles in the evaluation of the scientific authority of the journal. The findings of our research are presented in the form of an impact rating of economic journals, which reflects the level of their research impact, the importance and usefulness of their publications, and their scientific authority. The degree of correlation of the final list with other ratings of economic journals confirms that the criteria we propose and the method of their aggregation that we use enable to obtain fairly objective data suitable for ranking scientific journals.

In general, we can say that the integrated indicators obtained with the help of multidimensional comparative analysis of several significant bibliometric indicators of publications based on the data of the RSCI for 2016, allowed us to rank the journals and distinguish among them the core of the ten leading scientific journals in economics that are affiliated with academic organizations. These are well-known publications in the scientific community that have an impact on the development of economic science in the country.

It is obvious that the methodology for constructing the impact rating presented in this study can be used for ranking a wide range of scientific publications. In this case, the procedure for calculating some indicators, in particular the number of highly cited publications, would be simplified if the Russian Science Citation Index contained indicators that reflect the so-called “extreme” citation. A good solution to the problem of objective data collection would be to introduce tools that would automatically exclude links that have been manipulated. It is obvious that improving the systems that accumulate the initial data used in ranking will have a positive effect on the ranking results.

In conclusion we should note that ratings can be used as sources of additional information to analyze the effects and factors that hinder the success of academic publications, as well as reference points; they can help improve the quality of not only specific journals, but also the entire Russian economic science. As for the shortcomings that we have identified in the current ratings, they can be eliminated in the process of further study of the choice of criteria for ranking publications and improving the methodology of their analysis.