Meta-analysis of systematic literature review methods

Published Online February 2019 in MECS DOI: 10.5815/ijmecs.2019.02.03

Reports, surveys, interviews, direct observation and multiple other ways are frequently used to pull the information using different devices. To handle such a huge amount and diverse data present under diverse ways, many tools are implemented to analyze and integrate the most valuable data like [1], respecting some criteria and answering to some purpose to be able to process and manage Big Data.

This huge increase in data volume will have a critical impact on the overhead costs of computation, storage and networks [2].

As each study can influence and be a part of changing humanity life we take the same approach into research field. This need has been addressed individually within a number of disciplines like medicine, software engineering, education, biology and many others.

Systematic Literature Review (SLR) is a scientific approved method used since a long time ago [3], to collect all the data available respecting some predefined criteria to give an answer for a specific research question. Furthermore, to provide the best possible answer, different ways of comparing the quality of studies are established, to give a technique for the reader to judge the value of the work.

One of the criticisms that can be razed at researchers, is that they define the SLR method in some specific fields with no explicit definition. We, therefore, in this paper, relies on the methodology of SLR present in literature to conduct a complete Systematic Literature Review process with no domain condition in order to gain the greatest advantages from this technique. To extend the greatest important impact of this methodology especially in Medicine, the rest of this paper is organized as follows. Section 2 provides the importance of the use of systematic literature review as well as the benefit behind it, then the main reason of our paper. A background of this method is present in Section 3. Afterward in Section 4 the key steps raised according to a detailed approach are defined to conduct the structured steps. Lessons learned are then described and discussed in Section 5, and finally some conclusions are drawn.

• II.    Systematic Literature Review

Before focusing on the background of systematic literature review, we first need to understand the importance to establish a review and a systematic review.

• A.    The advantages of a Review

In a first appearance we can think the review is a waste of time with no real benefits. But, as described in [4] the literature review is not an exercise in style, but the essential element of the positioning of the research question, which is generally built up gradually and must be understood as a point of tension between knowledge and nonknowledge. Furthermore, [5] describe the importance of a review as: “Without a literature review, you will not be able to understand your topic fully. You will not be able to identify what has already been researched and what remains to be explored, and you will deny yourself valuable insights into those methods that are or are not appropriate for investigation of your topic. You will not only face the danger of reinventing the wheel but, even more critically, you will run the risk of ‘reinventing the flat tyre’!”. Those words show exactly the importance of a review, which can be used in any field for a good visibility and no redundancies work.

A review is a good approach to have the clear visibility about topics already discussed and go further on what they did discovered. It gives an overall idea about the subject in details to sustain a consistent work.

• B.    The advantages of a Systematic Review

Seven years ago, the article [6] confirm researchers using the Systematic review approach are increasing every day, with 11 published articles per day and no signs of slowing down.

Last year again a new research [7] display the exponential increasing researches tagged as “Systematic Review” as he presents Number of PubMed-Indexed Articles Published Each Year Between 1986 and 2014.

It seems very typical, to doubt in the importance of something referring only to the statistic of using it. Since, multiple reasons can be hidden behind it. However, many statements define clearly the gain like the article [8] who mentions the use of SLR on contribution for clinicians in a particularly service, who have a question based on their clinical practice and want to obtain the best evidence based answer. Also, under [9] mentioning that systematic reviews are already helping to identify “what works” beyond the world of evidence based medicine, and their potential role is more wide ranging than is often realized. In 2016 several researches shows the importance of SLR in evidence-based in Medicine on general as a high used approach.

Medicine is not the only discipline using SLR, but is one of the most important fields that impact human life in the first degree, and some decisions can lead to a big disaster if they are not well studied. “Systematic Review methods were developed, and have been employed, in healthcare for more than two decades, and they are now widely used across a broad range of topics, including environmental management and social interventions in crime and justice, education, international development, and social welfare” [10].

SLR provides an objective, transparent summary of the best available evidence and is designed to answer an a priori research question. It also allowed to identify the weaknesses in the literature used to generate a hypothesis for future research [11].

Multiple researches used systematic review to improve our life in different fields like [12] attested the side effect of computer games on users aged 14 years or above. Or, [13] medical article about new tracers for Prostate Cancer. Additionally, to increase the quality of teaching we can found the article [14] about the types of knowledge and skills essential to effectively teach online and critically examines how these elements are operationalized in current programs for training teachers to teach online.

No one can argue about the importance of an SLR in our life, but anyone can hesitate about the approaches to evaluate if the value behind it is fully respected or not, for a real benefit of it. In next section will discuss the background of this method for a better understanding of the lack of the steps described on SLR approach.

• C.    Motivation and Problematic

Systematic Literature Review (SLR) is defined in [15] as a type of literature review that collects and critically analyzes multiple research studies or documents through a systematic process. It has been defined in most of articles similarly even if they are specified in different subjects and domain.

In the book Systematic Approaches to a Successful Literature Review published on 2012 has a brief history of the first systematic review. It mention the value of systematic methods for identifying, extracting and appraising information from individual studies as a protection against biased interpretation of research was mention in 1753 by James Lind, the Scottish naval surgeon [5].

Light and Smith [16] were the first to propose bringing together all relevant original data from various research studies. Thus, thanks to Archie Cochrane who has attracted the attention of researchers on the SLR method and those since the year 1972 when he published his famous book [17] emphasizing the quality of published research and the importance to have an overall view before making decisions.

However, the quality of an SLR is questioned in many cases as for the article [16] evocate the existing of a duplicated systematic review, even if in [17] mention the need to clarify whether the planned systematic review has already been done or not before starting SLR. Therefore, misconceptions are still persist and increase in last years. Although, SR advocates, welcome constructive criticism and recognize its value in challenging view and developing innovative approaches, many of these criticisms are in fact misconceptions [9,10].

For those purposes we are conducting this research to minimize the misconceptions due to a lack of explicit general methodology assembling all disciplines.

The SLR is not only away to do a simple extract of researches responding to some criteria but it’s a method to extract some knowledge and decision on behalf of the results. The benefit behind SLR is very important and due to lack of well described strategy to follow as saw in literature the results can be misleading then lead to a disaster in some cases.

Our first goal in this paper is to present a full method to follow and gain confidence on the extracted results based on the different strategy given in literature.

• III.    Study of Slr Methods

Various terms used in the literature have been grouped together to define the method to follow in order to have a good quality of literature review. We first process by selecting all terms used in different articles with their definition for a better understanding for each step functionality. Then, we analyze the need of each functionality to improve the quality of the method as well as to give a clear and explicit way to follow. Next, we codify and group same meaning and purposes action on given a unique term. To conclude, we sorted all steps and present the final improved method.

We extract the steps name based on articles from 1971 to 2016 evoking the methodology SLR. In Table 1. we display the percentage of articles presenting the methodology of SLR per year based on the total of 54 articles.

Table 1. Number of coverage the methodology SLR from 1971 to 2016

Year	1971	1987	1993	1995	1996
N° articles	2	1	1	2	1
Percentage	3,70%	1,85%	1,85%	3,70%	1,85%

Year	1997	2000	2001	2002	2003
N° articles	1	2	1	4	2
Percentage	1,85%	3,70%	1,85%	7,41%	3,70%

Year	2004	2005	2006	2007	2008
N° articles	3	3	2	5	2
Percentage	5,56%	5,56%	3,70%	9,26%	3,70%

Year	2009	2010	2011	2012	2013
N° articles	5	3	3	5	2
Percentage	9,26%	5,56%	5,56%	9,26%	3,70%

Year	2014	2015	2016
N° articles	2	1	1
Percentage	3,70%	1,85%	1,85%

By analyzing the articles we can note the steps below:

A pre-planning of all concepts which we consider to include should be set before starting the systematic review. This step is very essential in SLR as indicated [18]. The plan outlines the question of the review and the rationale for the proposed methods to be used. It also includes details on how different types of studies will be located, evaluated and synthesized [19]. Describing the methods in advance is one way of minimizing a constant deviation from the truth, because once the results of the studies identified no modification of the way documents should be examined is allowed.

In the approach to improve SLR, [20] propose to have a training relative to systematic review. Subsequently, before undertaking a systematic review, it is necessary to identify the need for a review, to check if there is any existing tests or pending ones and if a further review is warranted [21]. Those point augur against redundancy.

A literature review is "systematic" if it is based according to [21] on a clearly formulated specific question(s), identify relevant studies, evaluates their quality and summarizes the evidence using an explicit methodology. Thus, a systematic review of literature is a way to identify, evaluate and interpret all available research pertinent to a specific question in a subject area or a phenomenon of interest. The second step after [22] is to extend the research question in a complete protocol, which will form the section methods to use for the final document.

The main strength to establish a protocol according to [23] is that it encourages critical to be explicit about how the review will be conducted. It helps the evaluator to consider the various stages of the process at the beginning of the review, to anticipate problems and bring a plan. A protocol is also a useful tool for promoting transparency, transferability and replicability. It describes what the evaluator intended to do and allows the review to be repeated later by others.

The development of a research strategy is an iterative process involving refinement based on some determination of the level of completeness achieved [21]. A description and justification of how specific research methods are combined to meet the desired level.

Data Management is presented in [22] in three stages: search and selection, process selection, and outcome evaluation. Thus, data collect has as interest in [21] to gather the most interesting results for the purpose of analyzing, discussing and learning. According to [21] various methods can be used for a better result during the evaluation phase of the study. [24] refers to the concept of commissioning, which is used both to solicit research group offers willing to undertake and to serve as a steering document for the advisory group to ensure that the article remains focused and relevant in the context.

According to [24] the absence of a systematic protocol means that unwise reader is unable to judge the completeness of the arguments made in such research.

The criteria for inclusion and exclusion of the study should flow directly from the review questions and be specified [22]. Thus, the reasons for inclusion and exclusion should be recorded. Conceptual taxonomy can facilitate this step [25] by positioning search terms or keywords in a context of higher terms (wider), subordinate (slighter) and coordinated (synonymous). [26] Consist to specify the linguistic constraints during the search strategy and contains a section that allows to plan the sources inclusion / exclusion on criteria to define.

It is very important to define the data evaluation criteria at the beginning [22] to avoid changing the criteria because of the results provided. Before grouping the data statistically, researchers often identify potential sources of differences between studies or heterogeneity [27]. The requirements for completeness depend from [26] by domain and research question. In order to find the relevant evidence, [28] considers that it is essential to define the appropriate procedures to use. The selection strategy is a crucial phase in the systematic literature review process [26].

The extraction strategy should describe according to [29] how the information required from each primary study would be obtained. If the data handling or require assumptions and inferences to be made, the protocol should specify a suitable validation process. The data management phase presents the core of the protocol following [27].

Among the specific features [26] that illustrate the systematic approach and improve the chances of providing the best evidence synthesized is data collect. The selection of study is as stated in article [21] by using the search string to query multiple databases and identify a set of candidate studies. Then, the researcher(s) using the criteria of inclusion and exclusion to eliminate candidates that are relevant studies, using the title first, second abstracts and full text third. During each iteration, researchers eliminate a candidate study only when it is clear that the study is irrelevant. The importance to establish data summary is indicated in article [18] as an essential step.

To ensure that all potential work is located, article [21] suggests additional research to improve the collect. In addition, the article focuses on the benefits of managing references to know how to predict duplicates. In this step it is necessary to establish a conclusion based on the data coming from the research by respecting the predefined protocol [28]. Article [26] discusses the notion of subgroup management, by defining a division criterion and keeping them at a limited number while assigning a keyword to each subgroup. The researcher can apply a number of criteria based on intra-study comparisons to distinguish subgroup analyzes that are credible from those that are not. It is prudent to pay attention and check carefully conflicting results that may be a result of incorrect generalization. A detailed analysis is recommended when performing this step which is interpretation [20].

The stage of data analysis is crucial according to [28] due to the number of information that can reveal. In spite of statistical analysis, depending on the chosen test the same set of data can be combined to give different conclusions. Hence the importance of sensitivity analysis [30]. Thus they justify the choice of an analysis of data to exclude according to inclusion / exclusion criteria. However, following Cochrane study [18] analysis of contradictory research must be established. Among the analysis steps we find also the comparison of data as indicated in [31]. In order to make a wise decision, data analysis is a subtle step that needs to be carefully addressed [32]. Data integration [31] allows for more in-depth analysis.

The critical evaluation of individual studies selected for inclusion is a crucial step in the review [31]. To evaluate quality criteria that defines the studies that can be considered valid for those to be considered invalid [28]. Evaluate the quality of the data included and extracted [30]. Finally, evaluate integration criteria that has been one of the subjects in the famous book [19]. During a synthesis actual results in each meta-analysis should be represented, including confidence intervals and coherence measures [32].

The article [20] states that systematic literature revues are different besides their qualities as well, which is displayed on the way of results are presented. Thus, the core of an analysis following [33] is a description of data obtained as size as well as various indices of central tendency and variability, thus allowing thanks to multiple method to deduce legitimate results. We also note that a derivation of the predefined protocol is possible in wellframed cases and a section where we determine the gaps and necessary explanations is added as mention in the article [34]. Six chapters are dedicated to the interpretation phase [35] to present the difficulty and importance of this step.

Although writing a summary for each finding proves a task that consumes time and energy but the gain is vast with data extraction eligible studies that you have identified. The basic notion here is that you have to extract and organize all the information from each article that you will need in order not to go back to the original article [33]. Drawing conclusions on the data summaries is considered [28] as a step in the systematic review process. The last phase following the book [36] is devoted to the recommendation to bring improvements.

In the same approach [37] refers to a section dedicated to future research that remains optional, as [26] which adds among the section the planning of future research and planning of future SLR in the possible and necessary cases. While in case of persisting problems they should be also mentioned [31] to provide a complete and a clear picture.

Publishing the results is the eighth and final step in the systematic review methodology presented in article [26]. Thus, as mentioned in article [24] after the synthesis of the data an update of the SLR can be established thus a good structure of the report is necessary for the understanding of the objectives and result of the research. The article [38] focuses on one crucial point which is how the researcher will write the report in a consistent and clear manner to the readers. An evaluation of the report by a specialized external team is suggested in article [24] to add more value to the article.

Various proposals have been mentioned in several articles like [39] to write a research calendar that will give a very clear vision of the procedures and methodology followed. Also specify publishing mechanisms used like what is present in the article [40].

Finally, a sections for bibliography and appendix can be mentioned explicitly as in article [33,38] or implicitly as in articles [41,42].

• IV.    Slr Method: The Key Steps

Several publications define the data collection methodology as a way to receive data as [43,44]. Also [41] allows to prepare questionnaires to collect data while being specific to a particular area, thus not allowing mutualization. We were able to identify only two articles presenting a methodology of data collection. [42] Established five steps to follow: establish the goals of the data collection;

develop list of questions of interest; establish data categories; design and test data collection form; collect and validate data. In last year only, another methodology not very different than the first one was defined by [43]: identifying data to be collected; defining data elements; observing or measuring values, or acquiring data by other means as in case of secondary data use; recording those observations and measurements; processing data to render them in electronic form if not in electronic format already and prepare them for analysis.

Through the study of different methods proposed in the literature, we come out with a set of steps that can build a fairly complete and detailed process for the creation of a Systematic Literature Review despite the field of research.

The various steps are as follow:

• I.    Pre-Planning

• 1.  A training relative to SR

• 2.  Identify the need of a review

• 3.  Specific question(s)

• 4.  Protocol

• 5.  Research strategy

• 6.  Data management

• 7.  Data collect

• 8.  Evaluation

• 9.  Commissioning

• II.    Protocol

• 1.    Inclusion / exclusion criterion

• a.    Identify keywords

• b.    Linguistic constraints

• c.    Sources inclusion / exclusion

• 2.    Data evaluation criteria

• 3.    Integration criteria

• 4.  Requirements

• 5.  Methods to use

• 6.  Procedures to use

• 7.  Selection strategy

• 8.    Extraction strategy

• III.    Data management

• 1.  Data collect

• a.    Selection studies

• b.    Data summary

• c.    Improve the collect

• d.    Managing references

• e.    Conclusion

• 2.    Subgroup management

• a.  Division criterion

• b.  Identify keywords

• c.    Subgroup analyzes

• d.    Interpretation

• 3.    Data analysis

• a.    Sensitivity analysis

• b.    Statistical analysis

• c.  Analysis data to exclude

• d.  Analysis contradictory research

• e.    Data comparison

• f.  Make a decision

• g.  Integration

• 4.    Evaluation

• a.   Evaluate quality criteria

• b.  Evaluate data quality

• c.   Evaluate integration criteria

• 5.  Synthesis

• IV.    Present results

• 1.  Data description

• 2.  Deduce results

• 3.  Determine the gaps

• 4.  Data interpretation

• 5.  Summary

• 6.  Conclusion

• 7.  Recommendation

• V.    Future research

• 1.  Define persisting problems

• 2.  Planning of future research

• 3.  Planning of future SLR

• VI.    Publish

• 1.   Structure the report

• 2.  Write the report

• 3.  Evaluate the report

• 4.  Project calendar

• 5.    Specify publishing mechanisms

• 6.  Bibliography

• 7.  Appendix

• V.    Results and Discussion

  While analyzing the articles presenting SLR method from 1971 to 2016, we observe many phases, which are essential, are omitted in some articles, not by lack of step in their methodologies, but by lack of detail.

In Table 2. we show the number of steps for each article mentioned, following increasing chronology: as we can perceive in general, at the beginning of the appearance of SLR methodology, the number of steps was increasing for short period. In 1995, the number of steps show the highest number comparing to all other years, attesting that all this years the methodology wasn’t enough detailed and some steps become too obvious to be written for some researchers.

In Fig.1. we certainly can confirm our vision of unclear steps in multiple research. For example the Future research phase is very important since it gives a link and a hint for further research to better improve any given SLR, but only 15% of articles are evoking this phase. In this figure we are giving a rate of articles including at least one of the steps present in each phase even if not all the steps are mentioned.

Our proposed methods contain 6 phases that includes in total 61 steps which is much higher than any of articles present in literature. We conclude our methodology based on 54 articles exposing SLR method from 1971 to 2016.

Table 2. Number of steps covered by article

[16]   (LIGHT, Richard & SMITH, Paul, 1971)	04
[31]  (FELDMAN & Kenneth A., 1971)	06
[30]   (SACKS S., et al., 1987)	19
[18]   (CHALMERS, Iain., 1993)	08
[26]   (Cook DJ, et al., 1995)	23
[33]  (ROSENTHAL, Robert, 1995)	10
[25]   (DUFF, Alistair, 1996)	06
[32]   (COOK, Deborah J., et al., 1997)	06
[44]  (JONES, Tina & EVANS, David, 2000)	09
[45]   (BADGER, D., et al., 2000)	08
[35]   (Egger M, et al., 2001)	11
[46]  (NEEDLEMAN, Ian G, 2002)	11
[47]   (CARR, Alan B, 2002)	10
[23]   (BOAZ, Annette, et al., 2002)	06
[27]   (BHANDARI, Mohit, et al., 2002)	14
[48]  (BIGBY, Michael & WILLIAMS, Hywel, 2003)	09
[49]   (KHAN, Khalid S., et al., 2003)	05
[36]   (ALDERSON, Phil, et al., 2004)	10
[29]  (KITCHENHAM, Barbara, 2004)	13
[50]   (DYBA, et al., 2007)	13
[51]   (CHEN, et al., 2009)	13
[52]   (Pai, et al., 2004)	08
[28]   (J. Biolchini, et al., 2005)	16
[53]  (WHITE, Adrian & SCHMIDT, Katja, 2005)	13
[54]   (MIAN, et al., 2005)	09
[55]   (Haynes, et al., 2006)	07
[19]  (PETTICREW & ROBERTS, 2006)	10
[24]   (Kitchenham & Charters, 2007)	19
[40]   (HALL, et al., 2012)	19
[56]   (BRERETON, et al., 2007)	13
[57]  (KITCHENHAM, et al., 2015)	13
[58]   (Jorge, et al., 2007)	07
[59]  (ARMSTRONG, et al., 2007)	10
[20]   (DIESTE, et al., 2008)	13
[38]   (CRONIN, et al., 2008)	08
[34]  (KITCHENHAM, et al., 2009)	09
[60]   (MOHER, et al., 2009)	06
[61]   (Denyer & Tranfield, 2009)	10
[62]   (Centre for Reviews and Dissemination, 2009)	13
[63]  (OKOLI & SCHABRAM, 2010)	05
[64]  (RYAN, 2010)	06
[37]   (SCIE, 2010)	11
[65]  (ZHANG & BABAR, 2011)	10
[66]   (JESSON, et al., 2011)	07
[67]  (UMAN, 2011)	12
[68]  (BETTANY-SALTIKOV, 2012)	13
[42]   (STAPIĆ, et al., 2012)	10
[69]  (COLICCHIA & STROZZI, 2012)	06
[39]   (BOOTH, et al., 2012)	11
[22]  (KITCHENHAM & BRERETON, 2013)	11
[70]   (CARVER, et al., 2013)	16
[71]   (HASSLER, et al., 2014)	16
[21]  (SIDDAWAY, 2014)	15
[15]  (CRUZ-BENITO, 2016)	07

0%       20%       40%       60%       80%      100%

Fig.1. Number of articles evoking at least one of the steps of each phase

• VI.   Conclusion

In this paper we put under spotlight the Systematic Literature Review methodology under analyze, since many miss understanding was raised. To provide this situation to become worse and losing faith on the high benefit of this method in research domain. We establish this first study of different steps present under literature. Knowing Medicine domain is one of the biggest area using SLR and relying on it in multiple critical decisions.

After a clarification of the use and significance of each steps analyzed, we codify the terms. Further, we proposed a complete methodology of SLR to be used as a guide. We maintain six basic Phases: Pre-planning to plan all process and evaluate the need of the study before beginning. A Protocol to define the rules to follow for having a clear and objective result. Data management to select and collect data based on predefined criteria. Present Result is the main objective of the review. Future Research suggestion if needed and relevant. The last phase is Publish the result founded.

This research can lead to a further work to establish an SLR of the benefit of using an SLR in all research domain and the high impact in human being life.