Journal rankings: theory, methodology, bibliometry

The reorganization of the Russian Academy of Sciences followed by the subordination of academic institutes to the Ministry of Science and Higher Education of the Russian Federation has led to the consolidation of the intervention of the paternalistic state in the scientific life of the country with the introduction of various formal indicators. All this has transformed into increasing demands on publication activity, the number of articles, and the “arithmetic of citation”. Against this background, the “Scientific Electronic Library eLIBRARY.RU” has become one of the most popular websites, and a very convenient place for the “game of figures”. The popularity of this Internet resource is actually due to the availability of opportunities to quickly compare the biometric indicators of journals for any sample and with a large number of indicators.

Critically assessing this trend, we would like to draw attention to the fact that citation indicators can be considered “proxy indicators” at best. We are talking about such indicators that are not directly related to the process under consideration, but serve as a kind of “substitute” for an unobservable or immeasurable variable. In fact, the existence of a link between the citation of an article and the quality of the published research result is only a hypothesis that is currently almost impossible to verify.

the RSCI on the subject “Economics. Economic index, Herfindahl index, aggregate rating, ranking,

We should also emphasize that bibliometric indicators based on citations and references lists do not always indicate that the authors are familiar with the works of other researchers and, unfortunately, do not reflect in any way the quality of articles and the level of journals in which they are published. As a confirmation to this thesis, we can cite the words of the Polish researcher S. Kozyr-Kowalski, who drew attention to pseudo-erudition, which promotes bureaucratic advancement, the practice of adding the works that the researcher did not read and often did not even hold in their hands, to the footnotes and references” (Kozyr-Kowalski, 1967, p. 35).

With this in mind, any attempts to use bibliometric indicators to clarify the “better–worse” ratio almost always suffer from subjective interpretation, for which they are regularly and reasonably criticized (Seglen, 1997; Adler et al., 2011; Waltman, 2016; Ideas and Numbers ..., 2016). Let us note once again that economists hold different opinions regarding citation and scientometrics in general. Some consider it an important assessment of scientific activity, while others criticize this “game of figures”. There is no definite answer here. The whole point is where and how this information resource is used.

This work presents the results of research on the problems of dissemination of knowledge within the framework of the current theory of the humanities sector of the paternalistic state. We are talking about science, more precisely, academic and university science, where knowledge is mainly produced, and about the system of its dissemination, the most important channel of which is represented by scholarly journals. This topic is endless and, clearly, it cannot be fully covered in one article, so it makes sense to immediately identify the problems that the article is devoted to. First of all, we want to discuss the place that participants in the processes of knowledge production and dissemination occupy in economic theory. Taking into account the clearly increased interest in citation, reinforced by the monetary motivation of publication activity, which increased the importance of scientometric ranking of journals, we consider it important to look into a number of theoretical and methodological issues of using this tool, including specific methods for constructing journal ratings and the allocation of certain samples of journals from their total population.

In this regard, the main role belongs to the information bases Web of Science, Scopus, RePEc and of course the RSCI, created in 2005 by the “Scientific Electronic Library eLIBRARY.RU”, where at the beginning of September 2023, more than 500 journals on the subject “Economics. Economic sciences” were registered. And although, after the well-known events, foreign databases have largely lost their importance, the authority of the journals registered in these databases remains high. The development of the RSCI quite naturally led to the allocation of narrower samples in this information base, defining the leading groups of journals – the RSCI (33 journals) and the Core (38 journals).

Based on the data from eLIBRARY.RU, another participant serving the processes of production, reproduction and dissemination of knowledge, the Higher Attestation Commission (VAK), determined its own selection of journals in the same database, which began to consider candidate and doctoral dissertations only if there are publications included in the list of journals from the VAK sample. The presence of these three samples – RSCI, RSCI Core, VAK list – which assess the level of the same journals in different ways raises an obvious question about the validity of each of them.

Taking this into account, let us formulate the main goals of this study: first, theoretical substantiation for the formation of the core of economic journals, based on constructive criticism of eLIBRARY.RU methodology and the basic principles of the “knowledge economics”1; second, critical analysis of a set of bibliometric indicators used by the RSCI in the methodology of ranking journals; third, algorithmic support for the construction of journal ratings based on the authors’ developments in the field of MW analysis.

Questions of methodology

It seems that one should start with analyzing the practice of forming the eLIBRARY.RU information base and the methodology for constructing the corresponding journal samples, according to which information on the citation of publications and their authors is collected in the RSCI. Constructing such an information base and relevant samples corresponds to the general trend of dividing the list of publications into several ordered groups. Foreign information databases, as is known, use the division into four quartiles, reflecting journal citation rate. In the Russian literature, various recommendations can be found considering ways to arrange the analyzed set of scholarly journals registered in the RSCI on the subject “Economics. Economic sciences” into several groups.

We should note that eLIBRARY.RU has moved away from simple digital solutions in its methodology, singled out a group of journals called the RSCI Core, and established, in accordance with the principles of determining the SCIENCE INDEX rating, that “all bibliometric indicators used in calculating the rating take into account citations only from the RSCI Core”2. At the same time, the Higher Attestation Commission, using the same bibliometric information of the RSCI, created its own list of academic journals.

Two samples. According to the methodology accepted by eLIBRARY.RU, the RSCI Core includes the leading journals represented in the RSCI, as well as publications that have passed the expert selection procedure and are represented in the Web of Science Core Collection international scientific citation information systems³. The advantages of such a decision include the use of meaningful criteria developed by an expert group, which simultaneously causes its well-known flaw, which can be formulated in the form of a traditional question: “And pray who are the judges?”

This traditional question cannot be answered by the name “working group” alone. Apparently, first, a transparent democratic procedure for selecting a group of experts in each scientific field and/or determining a representative sample from a statistically large array of specialists is required here; and then – a reasonable methodology for the examination itself. Otherwise, distrust is generated in the formation and expertise of such a group, and thereby in the subjective definition of a sample of journals called the RSCI Core.

The sample provided by the Higher Attestation Commission on the subject “Economics. Economic sciences” is divided into several ordered groups based on its own methodology4, which, as in the RSCI, contains two components: quantitative (bibliometric indicators) and qualitative (estimates of the expert group). At the same time, the list of scholarly journals was arranged in descending order of the “scientific significance rate” and divided into categories in the ratio: K1 – 25%, K2 – 50% and K3 – 25%5. Thus, the K1 category – the VAK version of the core – includes 85 journals on the subject “Economics. Economic sciences”.

In this case, the question remains: “And pray who are the judges?” Let us add three more comments. First, it turned out that the core of scientific journals according to the VAK version is more than twice as large as the RSCI Core. If we take into account that the Science Index ranking of journals takes into account the citation of articles only by the core, then the differences in the lists turn out to be very significant. Second, and perhaps most important, qualitative assessments are incomparable, because they are determined by different specialists, called “working” and “expert” groups. Third, obviously, this notorious “battle of the frogs and mice” (Rubinstein, 2023) must be ended. Even if we do not evaluate both approaches to determining the core of scientific journals in any way, we can draw a fairly simple conclusion about the need to create a unified group of specialists that contributes to solving the problem for both the RSCI and the Higher Attestation Commission.

Journals base. We proceed from the fact that when determining the appropriate sample of journals, a different approach should be used, which is not based on the subjective assessments of an unknown group of experts, but rather on formal motives that have a consistent basis. And first of all, it should be borne in mind that there is a connection between publication activity and the production of knowledge, which is mainly created in academic institutions and leading universities.

At the same time, research centers and institutes of the Russian Academy of Sciences, whose main task is to conduct fundamental research, act in two guises: as a kind of factory for the production of knowledge – 15 scientific organizations, and as publishers – 23 scientific journals that ensure the dissemination of knowledge and communication of scientists on the subject “Economics. Economic sciences” (Tab. 1). And it seems strange that most of these journals (13 out of 23) were not included in the eLIBRARY.RU sample, named the RSCI Core.

Table 1. Journals of institutes and research centers of the Russian Academy of Sciences on the subject “Economics. Economic sciences”

N1	24 journals	RSCI Core	Institutes and research centers of the Russian Academy of Sciences	N2
1	Ekonomicheskie i sotsial’nye peremeny: fakty, tendentsii, prognoz (Economic and Social Changes: Facts, Trends, Forecast)	No	RAS Vologda Research Center	1
2	Problemy razvitiya territorii (Problems of Territory’s Development)	No
3	Regional’nye issledovaniya	No	RAS Institute of Geography	2
4	Sovremennaya Evropa (Contemporary Europe)	Yes	RAS Institute of Europe	3
5	Problemy prognozirovaniya (Studies on Russian Economic Development)	Yes	RAS Institute of Economic Forecasting	4
6	Nauchnye trudy: Institut narodnokhozyaistvennogo prognozirovaniya RAN	No
7	Kontury global’nykh transformatsii: politika, ekonomika, pravo (Outlines of Global Transformations: Politics, Economics, Law)	Yes	Institute of Scientific Information for Social Sciences	5
8	Problemy rynochnoi ekonomiki (Market Economy Problems)	No	RAS Market Economy Institute	6
9	Problemy upravleniya (Control Sciences)	Yes	V.A. Trapeznikov Institute of Control Sciences	7
10	Region: ekonomika i sotsiologiya (Region: Economics and Sociology)	Yes	Institute of Economics and Industrial Engineering, Siberian Branch of RAS	8
11	Voprosy teoreticheskoi ekonomiki (Issues of Economic Theory)	No	Institute of Economics	9
12	Uroven’ zhizni naseleniya regionov Rossii (Living Standards of the Population in the Regions of Russia)	No
13	Vestnik Instituta ekonomiki Rossiiskoi akademii nauk (The Bulletin of the Institute of Economics of the Russian Academy of Sciences)	No
14	Ekonomika regiona (Economy of Regions)	Yes	Institute of Economics, Ural Branch of RAS	10
14	AlterEconomics	No
16	Prostranstvennaya ekonomika (Spatial Economics)	Yes	Institute for Economic Studies, Far Eastern Branch of RAS	11
17	Regionalistika (Regionalistics)	No
18	Sever i rynok: formirovanie ekonomicheskogo poryadka	Yes	Kola Science Center	12
19	Rossiya i novye gosudarstva Evrazii (Russia and New States of Eurasia)	No	Primakov National Research Institute of World Economy and International Relations	13
20	Mirovaya ekonomika i mezhdunarodnye otnosheniya (World Eсonomy and International Relations)	Yes
21	Ekonomicheskaya nauka sovremennoi Rossii (Economics of Contemporary Russia)	No	RAS Section of Economics	14
22	Ekonomika i matematicheskie metody (Economics and Mathematical Methods)	Yes	RAS Central Economics and Mathematics Institute	15
23	Tsifrovaya ekonomika	No
Source: own compilation.

Among them were the journal Economic and Social Changes: Facts, Trends, Forecast * issued by RAS Vologda Research Center, its English version is indexed in WoS; Economics of Contemporary Russia , the only journal issued by the Economics Department of the Russian Academy of Sciences; a number of other well-known publications, such as, for example, Regional Research of Russia issued by RAS Institute of Geography; and all the journals of one of the leading economic organizations – RAS Institute of Economics. As for the quality of academic journals that are not included in the RSCI Core, its assessments should be the result of appropriate research, rather than the a priori judgments of experts, who often defend only their own interests.

We believe that the very fact that the journal is published by a research center or institute of the Russian Academy of Sciences is a sufficient argument for its inclusion in the journal database that serves as the foundation for scientometric research, including journals citation and ranking. It is in these scientific organizations that the main research potential is concentrated, which makes it possible to ensure the necessary level of peer review and editorial preparation of articles. The basic set of analyzed journals on the subject “Economics. Economic sciences” should be determined taking into account what has been said above.

Subjecting the eLIBRARY.RU methodology to constructive criticism regarding the allocation of the journals core sample, we do not call for introducing drastic changes to it. Believing that much of the activities of the RSCI should be evaluated positively, we consider it advisable to expand the core to include the abovementioned 13 journals of research centers and institutes of the Russian Academy of Sciences.

At the same time, a meaningful analysis of the composition of the current RSCI Core makes it possible to identify five groups of journals: “Journals of RAS institutes” (Q₁), “University journals indexed in WoS or Scopus” (Q₂), “Journals of other publishers indexed in WoS or Scopus” (Q₃), “University journals not indexed in WoS or Scopus” (Q₄) and “Journals of other publishers not indexed in WoS or Scopus” (Q₅). Taking into account the knowledge production factor, which is created in academic institutions as well, a sixth group (Q₆) should be added to these five groups of journals – “Journals of RAS institutes not included in the RSCI Core” (Tab. 2–4) .

Table 2 shows that when the RSCI Core includes journals, the founders of which are RAS institutes (Q₆), the number of journals included in the general “journal base” (G) will be 51. We believe that this collection of journals – Baza.RU (Appendix 1) – should be the basis for any scientometric research and, we emphasize especially, for taking into account citations when calculating all bibliometric indicators used in ranking journals. It should also be noted that, as for the samples under consideration, it makes no sense to limit ourselves to journals included in the RSCI or the RSCI Core.

Thus, the Yadro.RU sample is no less interesting; this sample, instead of subjectively choosing two groups of journals (Q4 and Q5) issued by universities and other publishers, but not indexed in WoS and Scopus, could formally include in the analysis the journals of group (Q6) issued by RAS institutes. The RAN.RU sample is of particular interest, representing the union of two groups (Q1 and Q6). It includes all journals published by research centers and institutes of the Russian Academy of Sciences.

Comparing the RSCI Core with Yadro.RU, it is easy to see that they have a common part – the association of journals (Q1, Q2 and Q3), which presents scientific publications indexed in WOS and Scopus and which are founded by institutes of the Russian Academy of Sciences, universities and other publishers. Let us define this part of the “journal

Journals of RAS institutes not included in the RSCI Core (Q 6 group )	Journals included in the RSCI Core					Baza.RU (G)
	Indexed in WoS and Scopus			Not indexed in WoS and Scopus
	Journals of RAS institutes (Q1 group)	University journals (Q2 group)	Journals of other publishers (Q3 group)	University journals (Q4 group)	Journals of other publishers (Q5 group)
13	10	14	4	4	6	51

Tabl e 3. Quantitative characteristics of the RSCI Core sa mple

Journals of RAS institutes not included in the RSCI Core (Q 6 group )	Journals included in the RSCI Core					RSCI Core
	Indexed in WoS and Scopus			Not indexed in WoS and Scopus
	Journals of RAS institutes (Q1 group)	University journals (Q2 group)	Journals of other publishers (Q3 group)	University journals (Q4 group)	Journals of other publishers (Q5 group)
NO	10	14	4	4	6	38

Journals of RAS institutes not included in the RSCI Core (Q 6 group )	Journals included in the RSCI Core					Yadro.RU
	Indexed in WoS and Scopus			Not indexed in WoS and Scopus
	Journals of RAS institutes (Q1 group)	University journals (Q2 group)	Journals of other publishers (Q3 group)	University journals (Q4 group)	Journals of other publishers (Q5 group)
13	10	14	4	NO	NO	41

base” as G1= (Q1, Q2 and Q3). By introducing two more notations in sequence: G2 = (Q4 and Q5) and G3 = (Q6), we can see that the sample of journals in the RSCI Core = (G1 and G2), and the sample in Yadro.RU = (G1 and G6). In this case, the general journal base – Baza.RU – can be represented as G = (G1 and G2 and G3).

Bibliometric indicators. Let us start with the Declaration on Research Assessment (DORA) adopted in San Francisco in 2012, the first paragraph of which reads: “Do not use journalbased metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions”⁶. Unfortunately, this important declaration did not change anything and the journal impact factor calculated over one, two, three, four or five years remains the basis for virtually all journal rankings (Garfield, 1955).

Scopus uses three main indicators: “CiteScore” calculated by a 4-year impact factor; “SNIP” (Source-Normalized Impact per Paper) calculated using normalized values of journal impact factors; SCImago (SJR) which, in addition to the usual impact factor, uses a Google algorithm (Brin, Page, 1998). Without dwelling on the first two indicators, special attention should be paid to the SJR indicator that calculates the journal rating taking into account the weight of each of its articles; the higher the rating of the journal where this work is cited, the higher the weight of the article. In other words, by citing articles published in another journal, this journal, in fact, delegates its prestige to it ⁷.

The abovementioned indicator and the corresponding method for determining the rating of journals have both advantages and disadvantages. It is noteworthy that the calculation of the impact factor takes into account the different weight of the cited works. However, it seems necessary to be critical about the fact that in this case journals are ranked only by one criterion based on the average citation of articles. Therefore, the eLIBRARY. RU proposal to use several different indicators characterizing different citation metrics looks quite justified.

Thus, to determine the ranking of SCIENCE INDEX journals in the new RSCI methodology, it is proposed to use four bibliometric indicators: first, the “five-year normalized impact factor”, which characterizes the value of the average citation of articles in the journal; second, the “ten-year normalized Hirsch index of the journal”, reflecting the number of highly cited articles in the publication; third, the “average Hirsch index of authors of articles over the past three years”, reflecting the number of highly cited authors in the journal; this indicator, according to the RSCI, was specially calculated for this rating; fourth, the “average length of the text of articles over the past three years”, which, according to the authors of this methodology, “correlates well with the quality level of the journal based on expert assessment”. All indicators used in calculating the rating are normalized by dividing by the maximum value of the indicator in the journals sample8.

It must be said that the first three indicators do not cause much doubt; the specified set of indicators can be called the basic one for ranking journals. As for the fourth indicator, “average length of the text of articles over the past three years”, its inclusion in the methodology for calculating the rating of journals is at least surprising. It turns out that the longer the article, the higher the rating of the journal. Leaving this conclusion without comment, let us pay attention to the explanation given by the authors of the methodology: “This indicator turned out to be quite unexpected. Nevertheless, it consistently showed a good correlation with the quality level of the journal based on expert assessment”. We think that such an explanation indicates only the quality of the expert assessment, which leads to such an absurd result.

At the same time, the authors of the methodology considered it advisable that the indicators determining the overall rating of the journal should not include such an indicator of scientometric analysis as the Herfindahl index, used, as is known, to assess the degree of markets monopolization. In our case, we are talking about the possibility of correcting calculations taking into account the artificial improvement of the journals’ bibliometric indicators as a result of combining journals for crosscitation and/or increasing articles by employees of the founding organizations.

We agree with the authors of the methodology that an increased level of self-citation is often quite natural for specialized journals, in which most of the articles on a certain topic are concentrated. In such a situation, the Herfindahl index unreasonably lowers the rating of the journal with a sufficiently high quality of publications. And yet, given the fact that the majority of economic journals on the subject “Economics. Economic sciences” are broad-based publications, it seems appropriate to consider this bibliometric indicator as a possible addition to the basic set of indicators that determine the overall rating of journals.

On the methodology for determining journal ratings. Let us first consider the methodology for calculating the rating of journals proposed by the RSCI working group. According to its authors, the final result of the journals ranking – their cumulative rating – can be determined as a linear combination of selected bibliometric indicators with an appropriate set of their weights. As the authors of this methodology point out, “the problem lies in the fact that the choice of both the composition of the indicators involved in the formula and their weight coefficients was carried out purely intuitively and was not confirmed biometrically in any way”9. We understand the authors of the methodology, who expressed their doubts about its correctness. Indeed, it is difficult to count on the correct result with an arbitrary (intuitive) choice of the weight function.

The correct determination of these weights is not an easy task when constructing any integral indicator. And yet, the solution chosen by the authors of the methodology cannot be called acceptable in any way. In fact, we are talking about an adjustment, though it is veiled: when from various combinations of the desired weights, presumably belonging to the range from 0 to 10, one was selected that showed a large correlation of the calculated rating with the rating of public expertise10. Such a solution has obvious disadvantages. First, if the calculated bibliometric rating is adjusted depending on its proximity to the rating of public expertise, then a natural question arises, why is bibliometrics needed at all? Second, the irrelevance of the public expertise itself, which was carried out five years ago, is confusing.

In general, the ranking of journals is based on solving the well-known problem of folding a set of parameters into a single criterion. Without repeating the well-known criticism of existing approaches to aggregating particular indicators, when, in addition to the analyzed RSCI methodology, one proceeds from the hypothesis of equality of weights or considers their random set, we will use the previously created methodology for assessing the contribution of individual biometric indicators to the aggregate rating.

We are talking about the application of MW analysis (Multiway data analysis), presented in the work (Rubinstein, Slutskin, 2018) and repeatedly tested in relation to various objects (Burakov et al., 2019; Burakov, Rubinstein, 2020; Burakov, 2021). It should also be noted that this method is a generalization of factor analysis, including the principal component analysis, in relation to a multidimensional matrix (tensor).

10 As a result of such manipulations, the RSCI working group settled on the following set of weights: for the impact factor for 5 years – 8, for the Hirsch index of articles in the journal for 10 years – 7, for the Hirsch index of authors of articles in the journal for 3 years – 4, for the average length of the text of articles for 3 years – 4.

See:

In this study, we are talking about the trivalent tensor V = {V_ij_g}. It forms an information parallelepiped containing 204 numbers, each of which can be represented by three coordinates: projections on the axis of the totality of analyzed journals i ∈ [1, 51]; on the axis of bibliometric indicators j ∈ [1,4] – five-year impact factor [j =1], ten-year Hirsch index of the journal [j = 2], average Hirsch index of authors of articles over the past three years [j = 3], Herfindahl index [j = 4]; on the axis of samples from the general journal database g ∈ [1,3] – groups of journals G₁, G₂, G₃.

The main idea of the MW analysis is related to the restoration of the original tenor characterizing the abovementioned three-dimensional space by means of its representation as an external product of three vectors of different dimensions, which can be interpreted as an aggregate rating of journals V_i ; the weights in this rating of individual bibliometric indicators Vj and the contributions to this rating of individual groups of journals Vg characterizing the corresponding components of the RSCI Core and Yadro.RU.

Let us consider the results of the MW analysis for two calculation options. In the first version, three bibliometric indicators were used, corre-ponding to the RSCI methodology (with the exception of the “average length of the text of articles over the past three years”); in the second version, another indicator was added – “Herfindahl index for authors’ organizations”. The results of these calculations for the general array of journals ( G ) are shown in Tables 5 and 6 .

The calculation results (option I) indicate that the “Hirsch index of authors of articles” demonstrates the strongest influence, and the impact factor makes the least contribution to the aggregate rating of journals. When using an additional indicator, the Herfindahl index (option II), the ratios remained the same, but the contribution of each of the three basic indicators decreased slightly, “transferring” part of its influence to the fourth indicator.

Table 5. Contribution of individual indicators to the aggregate rating of journals, %

Rating option	Weights in the aggregate rating
	5-year impact factor	10-year Hirsch index	Average Hirsch index for 3 years	Herfindahl index
I	24.2	30.5	45.3
II	20.9	26.5	41.8	10.9

Table 6. Contribution of individual samples of journals to their aggregate rating, %

Rating option	Weights in the aggregate rating
	Sample G1	Sample G2	Sample G3
I	45.5	26.4	28.1
II	35.6	34.6	29.8

In addition we should point out that the idea of including the Hirsch indices in a set of indicators based on which the rating of journals is calculated is a definite innovation. This is the first time that this scientometric indicator has been used to rank scholarly journals. And despite all the failures of the new RSCI methodology, the three selected bibliometric indicators deserve the attention of specialists. At the same time, we note that the results of applying the MW analysis procedure for a set of indicators not only indicate the incorrectness of the RSCI methodology using a priori set weights, but, most importantly, refute the ingrained idea that the main indicator in the formation of journal ratings is the impact factor, that is, the average number of citations of one article for a certain period.

This conclusion proceeds from a number of considerations. First of all, they are associated with a chronic lack of any average values that do not take into account the nature of the distribution of the analyzed indicator. In this sense, the Hirsch index, which is well known in scientometrics, compares favorably with the impact factor. Moreover, in recent years it has become the most popular for “evaluating the scientific performance of individual scientists, research teams and organizations” (Nazarenko, 2013). And, as the calculations have shown, higher values of the Hirsch indices increase the rating of journals with highly cited articles and articles by highly cited authors.

Table 6 shows equally interesting results. The calculations performed prove that with three bibliometric indicators (option I), the group of journals G3 that are part of Yadro.RU has a greater impact on the aggregate rating than the journals from sample G2 included in the RSCI Core. Taking into account the fact that both cores of economic journals have the same common part G1, it is possible to make a statistically substantiated conclusion about the greater informational significance of Yadro.RU in comparison with the RSCI Core (see Tab. 5). This result is an additional argument for a fundamental conclusion about the expediency of using Yadro.RU journals in the scientometric ranking, because Yadro.RU is based on meaningful criteria that take into account the links between publication activity and knowledge generation.

The results of MW analysis are also very remarkable for another calculation option, due to the introduction of an additional indicator – the Herfindahl index. In this case, the situation is reversed: group of journals G3 that are part of Yadro.RU has less impact on the overall rating than journals from G2 sample (option II). In other words, if the ranking of journals on the subject “Economics. Economic sciences” is based on an aggregate rating that uses four bibliometric indicators, taking into account the Herfindahl index, then the RSCI Core is more informative.

Given the role of this indicator, which, as we already noted, consists in assessing the decline in the importance of journals with a high proportion of self-citations or co-citations, these differences are easy to explain. Apparently, such influence of the Herfindahl index has the greatest effect on journals founded by scientific centers and institutes of the Russian Academy of Sciences.

The fact that the state assignment combines the substantive aspects of research related to the production of knowledge with the number of journal publications related to its dissemination leads to an increase in the publication of articles in journals by the own staff of academic institutions. As an example, let us name the journal Regionalistics, in which in 2022 75% of all articles were published by employees of the journal’s founding institute. Based on the results of the performed analysis, several conclusions can be drawn.

First, the core of the journals should not stand out from their total array, regardless of the set of indicators that determine the aggregate journal rating. Second, if the purpose of ranking journals is to determine their authority solely for reasons of citation, then the most informative is Yadro. RU. Third, if one wants to “revise” the citation of journals and adjust their impact estimates on this basis, the RSCI Core is more useful.

Ranking scholarly journals

The results obtained allow us to conclude that the use of MW analysis not only changes the ratio of contributions of bibliometric indicators to the aggregate rating in relation to the RSCI methodology, but, equally importantly, changes the ranking of journals themselves. We should note that, in our opinion, the very procedure for selecting journals that the expert group has included in the core is insufficiently substantiated and, therefore, vulnerable.

The RSCI Core journals. Using the weights of bibliometric indicators obtained as a result of the application of MW analysis (see Tab. 5), and the corresponding rating of journals, the obtained ranking options can be compared with two variants of journal ratings calculated on the basis of the RSCI methodology – the old (as of January 9, 2023) and the new (as of October 24, 2023) variants of the SCIENCE INDEX11. The results of this comparison are presented in Table 7, which contains three ranking options for scholarly journals included in the RSCI Core.

The analysis of Table 7 allows us to identify several journals that, according to MW analysis, have improved their position in the RSCI Core compared with the accepted methodology. Let us name, for example, the following journals: Journal of Institutional Studies (by 2 and 8 points), Economics and Mathematical Methods (for 8 and 9 points), Region: Economics and Sociology (by 2 and 6 points), St Petersburg University Journal of Economic Studies (by 2 and 12 points). We should also note the journals whose positions according to MW analysis have deteriorated in the RSCI Core: Economy of Regions, World Economy and International Relations, Foresight and STI Governance, Financial Journal, Economic Policy and Business Informatics. As for other journals included in the RSCI Core, the changes were insignificant.

Comparing the ranks of journals corresponding to the old and new versions of Science Index, we see that they arouse a certain distrust regarding the results of RSCI calculations; especially striking is the sharp change in the position of Foresight and STI Governance, which literally in one month plummeted from the 6th to the 31st place. It is clear that in such a situation any subjective decisions in the construction of this core of journals and flaws in the methodology of calculations of their ranking can lead to negative consequences for the entire system of publication activity and, importantly, for the current system of incentives for researchers. Next, let us consider a sample of journals based on their connection with the processes of knowledge generation.

Table 7. Ranking of scientific journals included in the RSCI Core

No.	JOURNAL	RSCI methodology (Science Index, option I)	RSCI methodology (Science Index, new option)	MW analysis (4 indicators, option III)
13	Voprosy ekonomiki	1	1	1
21	Problemy prognozirovaniya (Studies on Russian Economic Development)	2	2	2
5	Russian Journal of Economics	5	8	3
15	Zhurnal Novoi ekonomicheskoi assotsiatsii (Journal of the New Economic Association)	8	9	4
36	Ekonomika regiona (Economy of Regions)	3	3	5
3	Journal of Institutional Studies	14	13	6
6	Terra Economicus	9	10	7
34	EKO (ECO)	7	7	8
23	Prostranstvennaya ekonomika (Spatial Economics)	10	18	9
18	Mirovaya ekonomika i mezhdunarodnye otnosheniya (World Eсonomy and International Relations)	4	4	10
24	Region: ekonomika i sotsiologiya (Region: Economics and Sociology)	17	15	11
33	Forsait (Foresight and STI Governance)	6	31	12
26	Rossiiskii ekonomicheskii zhurnal (Russian Economic Journal)	15	17	13
16	Kontury global’nykh transformatsii: politika, ekonomika, pravo (Outlines of Global Transformations: Politics, Economics, Law)	12	12	14
28	Sovremennaya Evropa (Contemporary Europe)	19	11	15
22	Problemy upravleniya (Control Sciences)	18	6	16
37	Ekonomicheskaya politika (Economic Policy)	13	16	17
35	Ekonomika i matematicheskie metody (Economics and Mathematical Methods)	27	19	18
9	Vestnik mezhdunarodnykh organizatsii: obrazovanie, nauka, novaya ekonomika (International Organisations Research Journal)	11	21	19
8	Vestnik MGIMO Universiteta ( MGIMO Review of International Relations)	20	14	20
29	Universitetskoe upravlenie: praktika i analiz (University Management: Practice and Analysis)	23	24	21
27	Sever i rynok: formirovanie ekonomicheskogo poryadka	29	26	22
30	Upravlenets (The Manager)	30	28	23
38	Ekonomicheskii zhurnal Vysshei shkoly ekonomiki (The HSE Economic Journal)	16	22	24
20	Prikladnaya ekonometrika (Applied Econometrics)	21	23	25
12	Vestnik Sankt-Peterburgskogo universiteta. Ekonomika (St Petersburg	38	36	26
	University Journal of Economic Studies)
32	Finansy: teoriya i praktika (Finance: Theory and Practice)	28	25	27
4	Journal of Tax Reform	32	32.	28
19	Prikladnaya informatika (Journal of Applied Informatics)	25	29	29
3 1	Finansovyi zhurnal (Financial Journal)	24	20	30
7	Biznes-informatika (Business Informatics)	26	5	31
2	Journal of Applied Economic Research	33	33	32
25	Rossiiskii zhurnal menedzhmenta (Russian Management Journal)	36	34	33
14	Den’gi i kredit (Russian Journal of Money and Finance)	22	27	34
17	MIR (Modernizatsiya. Innovatsii. Razvitie) (MIR (Modernization. Innovation. Research))	31	30	35
1	Ars Administrandi	34	37	36
10	Vestnik Moskovskogo universiteta. Seriya 6: Ekonomika (Moscow University Economic Bulletin)	35	35	37
11	Vestnik Sankt-Peterburgskogo universiteta. Menedzhment (Vestnik of Saint Petersburg University. Management)	37	38	38
Source: own compilation.

The Yadro.RU journals . We recall that this sample, in addition to journals representing academic science (Q₁ and Q₆), includes university journals directly involved in knowledge generation (Q₂), and journals of other publishers (Q₃) whose indexing in foreign databases Web of Science Core and Scopus is the criterion for their inclusion in Yadro.RU. Thus, 23 academic journals, 14 university journals and 4 journals of other publishers were included in Yadro.RU. The total number of academic journals amounted to 41.

As in the case with the RSCI Core, we present the results of the ranking of scientific journals included in Yadro.RU (Tab. 8). Comparing the rankings of journals in Yadro.RU and the RSCI Core calculated using the same MW analysis procedure, it is easy to see that the ranks of journals included in both Yadro.RU and the RSCI Core do not have big differences. We should note that 6 out of 11 journals included in the first quartile Q1 of the aggregate journal database Baza.RU are publications of scientific centers and institutes of the Russian Academy of Sciences. At the same time, we cannot ignore another fact: 6 out of 10 journals included in the fourth quartile Q4 of the aggregate journal database, represent institutes and scientific centers of the Russian Academy of Sciences with low ratings, which occupy positions from 32 to 41 in Yadro.RU.

As a conclusion to this article, we considered it advisable not only to point out critical comments regarding the methodology used in the RSCI, but also formulate specific recommendations related to the activities of the “Scientific Electronic Library eLIBRARY.RU”. The elimination of a number of identified shortcomings in the ranking methodology used for scholarly journals is not only relevant from the point of view of adequately determining their rating, but, more importantly, it may somewhat mitigate the situation with the imposed “publication race” for academic and university scientists involved in knowledge generation. The recommendations can be formulated as follows.

No.	JOURNAL	Founder	Part of the RSCI Core	Quartile in Baza.RU	JOURNAL RANK
					BAZA.RU	RSCI CORE	YADRO.RU
1	Voprosy ekonomiki	Other	-	Q1	1	1	1
2	Problemy prognozirovaniya (Studies on Russian Economic Development)	RAS	YES	Q1	2	2	2
4	Journal of Institutional Studies	Other	-	Q1	4	4	3
5	Zhurnal Novoi ekonomicheskoi assotsiatsii (Journal of the New Economic Association)	Other	-	Q1	5	5	4
6	Russian Journal of Economics	Other	-	Q1	7	7	5
7	Ekonomika regiona (Economy of Regions)	RAS	YES	Q1	6	6	6
8	Terra Economicus	Univer.	-	Q1	8	9	7
9	Ekonomicheskie i sotsial’nye peremeny: fakty, tendentsii, prognoz (Economic and Social Changes: Facts, Trends, Forecast)	RAS	-	Q1	9	-	8
11	Prostranstvennaya ekonomika (Spatial Economics)	RAS	YES	Q1	11	10	9
10	Mirovaya ekonomika i mezhdunarodnye otnosheniya (World Eсonomy and International Relations)	RAS	YES	Q1	10	8	10
13	Kontury global’nykh transformatsii: politika, ekonomika, pravo (Outlines of Global Transformations: Politics, Economics, Law)	RAS	YES	Q1	13	13	11
1 4	Forsait (Foresight and STI Governance)	Univer.	-	Q2	14	11	12
15	Regional’nye issledovaniya	RAS	-	Q2	15	-	13

End of Table 8

No.	JOURNAL	Founder	Part of the RSCI Core	Quartile in Baza.RU	JOURNAL RANK
					BAZA.RU	RSCI CORE	YADRO.RU
16	Region: ekonomika i sotsiologiya (Region: Economics and Sociology)	RAS	YES	Q2	16	14	14
17	Sovremennaya Evropa (Contemporary Europe)	RAS	YES	Q2	17	15	15
18	Ekonomicheskaya politika (Economic Policy)	Univer.	-	Q2	18	16	16
19	Problemy upravleniya (Control Sciences)	RAS	YES	Q2	19	17	17
20	Ekonomika i matematicheskie metody (Economics and Mathematical Methods)	RAS	YES	Q2	20	18	18
24	Voprosy teoreticheskoi ekonomiki (Issues of Economic Theory)	RAS	-	Q2	24	-	19
22	Vestnik mezhdunarodnykh organizatsii: obrazovanie, nauka, novaya ekonomika (International Organisations Research Journal)	Univer.	-	Q2	22	19	20
25	Nauchnye trudy: Institut narodno-khozyaistvennogo prognozirovaniya RAN	RAS	-	Q2	25	-	21
23	Biznes-informatika (Business Informatics)	Other	-	Q2	23	21	22
29	Sever i rynok: formirovanie ekonomicheskogo poryadka	RAS	YES	Q3	29	27	23
28	Upravlenets (The Manager)	Univer.	-	Q3	28	25	24
31	Uroven’ zhizni naseleniya regionov Rossii (Living Standards of the Population in the Regions of Russia)	RAS	-	Q3	31	-	25
30	Finansy: teoriya i praktika (Finance: Theory and Practice)	Univer.	-	Q3	30	26	26
33	Ekonomicheskii zhurnal Vysshei shkoly ekonomiki (The HSE Economic Journal)	Univer.	-	Q3	33	24	27
34	Ekonomicheskaya nauka sovremennoi Rossii (Economics of Contemporary Russia)	RAS	-	Q3	34	-	28
36	Journal of Tax Reform	Univer.	-	Q3	36	31	29
37	Prikladnaya ekonometrika (Applied Econometrics)	Univer.	-	Q3	37	30	30
38	AlterEconomics	RAS	-	Q3	38	-	31
40	Regionalistika (Regionalistics)	RAS	-	Q4	40	-	32
41	Rossiiskii zhurnal menedzhmenta (Russian Management Journal)	Other	-	Q4	41	33	33
43	Vestnik Instituta ekonomiki Rossiiskoi akademii nauk (The Bulletin of the Institute of Economics of the Russian Academy of Sciences)	RAS	-	Q4	43	-	34
44	Problemy razvitiya territorii (Problems of Territory’s Development)	RAS	-	Q4	44	-	35
46	Vestnik Moskovskogo universiteta. Seriya 6: Ekonomika (Moscow University Economic Bulletin)	Univer.	-	Q4	46	37	36
48	Problemy rynochnoi ekonomiki (Market Economy Problems)	RAS	-	Q4	48	-	37
47	Ars Administrandi	Univer.	-	Q4	47	36	38
49	Tsifrovaya ekonomika	RAS	-	Q4	49	-	39
50	Vestnik Sankt-Peterburgskogo universiteta. Menedzhment (Vestnik of Saint Petersburg University. Management)	Univer.	-	Q4	50	38	40
51	Rossiya i novye gosudarstva Evrazii (Russia and New States of Eurasia)	RAS	-	Q4	51	-	41
Source: own compilation.

• 1.    It is necessary to create a common array of the journals – Baza.RU – which includes 51 academic journals. At the same time, for analytical purposes, it makes sense to keep the samples such as the RSCI Core, Yadro.RU and RАN.RU.

• 2.    The Higher Attestation Commission should be proposed to establish the K1 category for all scholarly journals included in Baza.RU.

• 3.    The citation of publications, which is appropriately reflected in bibliometric indicators, should be recorded in the RSCI only if the article is cited in a journal that is part of the aggregate journal database Baza.RU.

• 4.    Considering the practical importance of taking into account citation and “purified” citation when ranking journals, it is advisable to use both versions of bibliometric indicators. We are talking about a basic set that includes a five-year impact factor, ten- and three-year Hirsch indices, as well as an expanded set of indicators supplemented by the Herfindahl index.

• 5.    It is necessary to support the proposal by V. Glukhov, Deputy Director General of the Scientific Electronic Library eLIBRARY.RU, about uploading reviews of articles to the RSCI system together with publications, which, in addition to public evidence of peer review, can improve the ongoing examination of journals, making it less subjective and more professional.

• 6.    It is necessary to make fundamental changes to the methodology for determining the overall rating of journals. We are talking about the application of MW analysis, which, using real data contained in the information base of the “Scientific Electronic Library eLIBRARY.RU” allows identifying the statistically substantiated contribution of each bibliometric indicator to the overall rating, as well as calculating the value of this rating for each journal.

• 7.    It is advisable to establish a procedure according to which the journals ranking results should be reviewed every year (no later than February 1) on the basis of up-to-date information from the “Scientific Electronic Library eLIBRARY.RU”.