Serviços Personalizados
Journal
Artigo
Compartilhar
Revista Brasileira de Educação
versão impressa ISSN 1413-2478versão On-line ISSN 1809-449X
Rev. Bras. Educ. vol.29 Rio de Janeiro 2024 Epub 27-Nov-2024
https://doi.org/10.1590/s1413-24782024290111
Article
The integrative review in studies of educational public policy: potential and applicability of the method
ELDARONICE QUEIROZ DE ALVARENGA has a doctorate in education from the Universidade Federal de MInas Gerais (UFMG). She is an educational analyst at the State Secretariat of Education of Minas Gerais. Director at the Regional Superintendece of Education of Pirapora-MG (SEE-MG).
, Writing - original draft, Conceptualization, Methodology, Data curation, Formal analysis, Funding acquisition, Research, Project administration, Resources, Software, Validation, Visualization, Writing - review & editing
http://orcid.org/0000-0002-0833-422X
MARIA CLARICE LIMA BATISTA has a doctorate in social psychology from the Universidade Federal de Minas Gerais (UFMG). She is a librarian in the Researcher Support Sector of the Capes Portal of the same institution.
, Research, Formal analysis, Supervision, Validation, Visualization, Writing - review & editing
http://orcid.org/0000-0001-5907-0095
EYLEEN NABYLA ALVARENGA NIITSUMA has a doctorate in nursing from the Escola de Enfermagem da Universidade Federal de Minas Gerais (UFMG). She is a nursing professor at the Instituto Federal de Educação do Norte de Minas Gerais (IFNMG).
, Conceptualization, Formal analysis, Methodology, Software, Validation, Visualization, Validation, Writing - review & editing
http://orcid.org/0000-0002-5781-6313
ROSIMAR DE FÁTIMA OLIVEIRA has a doctorate in education from the Universidade de São Paulo (USP). She is the coordinator of the Postgraduate Program in Education: Knowledge and Social Inclusion, at the Faculty of Education of the Universidade Federal de Minas Gerais (UFMG). She is also the coordinator of the research group on policy and administration of educational systems (Pase) at the same institution.
, Conceptualization, Formal analysis, Funding acquisition, Project administration, Resources, Supervision, Validation, Writing - review & editing
http://orcid.org/0000-0002-2212-4018
ISecretaria de Estado de Educação de Minas Gerais, Pirapora, MG, Brazil. Email: eldaroniceqalvarenga@gmail.com
IIUniversidade Federal de Minas Gerais, Belo Horizonte, MG, Brazil. Email: clarycelima@gmail.com
IIIInstituto Federal de Educação do Norte de Minas Gerais, Almenara, MG, Brazil. Email: eyleenmg@hotmail.com
IVUniversidade Federal de Minas Gerais, Pampulha, MG, Brazil. Email: rosimarfoliveira@gmail.com
This article aims to present the possibilities of using integrative review as a method that analyzes and synthesizes literature reviews in the field of education, and also to present a protocol that shows how it was possible to apply this methodology in a study of educational public policy. The investigation sought to answer whether the integrative review method is applicable in literature reviews in research conducted in this field. To this end, the narrative review technique was used, with mapping of the review from a qualitative perspective. The investigation points out that integrative review, when supported by the use of protocols appropriate to the research object, constitutes a method applicable to research in the field of educational policies.
Keywords: Educational Public; Integrative Review; Integrative Review Protocol
Este artigo tem como objetivo apresentar as possibilidades do uso da revisão integrativa como método que analisa e sintetiza revisões de literatura no campo da educação e, também, propor um protocolo que demonstra a possibilidade de aplicar essa metodologia em um estudo das políticas públicas educacionais. A investigação buscou responder se o método da revisão integrativa é aplicável nas pesquisas realizadas nesse campo. Com esse fim, utilizou-se a revisão narrativa com mapeamento da literatura na perspectiva qualitativa. A investigação mostra que a revisão integrativa, quando apoiada em protocolos apropriados ao objeto da pesquisa, constitui-se como um método aplicável às pesquisas do campo das políticas educacionais.
Palavras-chave: Política Educacional; Revisão Integrativa; Protocolo de RI
Este artículo tiene como objetivo resaltar las posibilidades de utilizar la Revisión Integrativa como un método que analiza las literaturas en el campo de la Educación, y también proponer un protocolo que demuestra cómo es posible aplicar esta metodología en un estudio de políticas públicas educativas. La investigación buscó responder si el método de Revisión Integrativa es aplicable en revisiones de literatura en investigaciones realizadas en este campo. Para ello se utilizó la Revisión Narrativa, mapeando la revisión desde una perspectiva cualitativa. La investigación apunta que la Revisión Integrativa, cuando apoyada en el uso de protocolos adecuados al objeto de investigación, constituye un método aplicable a la investigación en el campo de las políticas educativas.
Palabras clave: Política Educativa; Revisión Integradora; Protocolo de Revisión Integrativa
INTRODUCTION
Research in education in Brazil gained prominence within universities with the establishment of postgraduate programs in 1965, followed by significant expansion during the 1970s, particularly through the creation of research groups (Vosgerau and Romanowski, 2014). This growth was further driven by an increase in academic production in the field, partly due to the requirements of the National Postgraduate System and the growing importance of research results in shaping public educational policies (Gatti, 2012). Consequently, scientists are increasingly required to conduct investigations based on literature reviews, with the aim of mapping, analyzing, and synthesizing research published across various fields of education.
The literature review seeks to problematize emerging investigative demands and highlight methodological designs that can be applied across various sub-areas of knowledge (Vosgerau and Romanowski, 2014). Methodologically, the review process necessitates studies from diverse bibliographic sources to enhance understanding of a particular topic, serving as the foundational stage in the construction of scientific knowledge. Within this framework, research that relies on literature reviews, either as the sole investigative source and/or as an initial phase, focuses on conducting a comprehensive analysis and a rigorous synthesis to expand and deepen scientific understanding of the subject under study. This approach positions the literature review as the first step in the development of scientific knowledge (Ingram et al., 2006).
Various methods exist for conducting a literature review. These include techniques such as the traditional bibliographic review, also known as the narrative review (NR), as well as the use of mechanisms and methodologies associated with systematized literature reviews (SLR),1 with the integrative review (IR) being particularly notable. The spread of diverse research methods for organizing and discussing academic studies has led to a selection of more systematic and rigorous literature review (LR) methods, as highlighted by Whittemore and Knafl (2005).
From this perspective, the primary objective of this article was to explore the potential of using IR in LR research within the field of public educational policies. To achieve this, the text discusses the IR method as an SLR approach, emphasizing its applicability for academics and researchers in education. Additionally, it presents a protocol, supported by evidence,2 demonstrating how this method was effectively applied in a study focused on the subarea of public educational policies.
Following this introduction, the text is organized into three sections. The first outlines the methodological framework of the studies that form the basis of this article. The second offers a theoretical reflection on the LR process, examining its conceptual and methodological implications and problematizing its various approaches, types, and methods, setting the context for discussing the IR method. The third section focuses on the application of the IR method in the field of educational policies, detailing the step-by-step process and criteria used to develop an IR Protocol for conducting an SLR as part of a research project at the School of Education at the Universidade Federal de Minas Gerais (Alvarenga, 2019). The article concludes with final considerations.
The study's findings indicate that the IR, when supported by protocols tailored to the research subject, is a viable method for research in the field of educational policies, which is rapidly expanding in Brazil. The investigation also revealed that the limited publication of methods and protocols used in LR within the realm of public educational policies positions the results presented here as having significant theoretical potential for the field. Consequently, this text contextualizes methodologies predominantly used in the health sector and adapts them for application in studies of educational policies.
METHODOLOGICAL DESIGN
Based on bibliographic studies on LR in the educational field, it has been observed that scholars frequently highlight the weaknesses of review studies and the need for greater methodological refinement in this area of knowledge (Gough, 2007; Thomas, 2007; Vosgerau and Romanowski, 2014). These concerns prompted us to take on the challenge of mastering LR techniques to enhance the rigor of review studies in the field of public educational policies.
Thus, the IR, the focus of investigation in this work, emerged from a problem discussed and reflected upon during doctoral research: Is the use of IR, as an SLR method, suitable for literature reviews in the field of public policies? If so, is there an existing IR protocol in this field of knowledge that could be utilized in our research? To address these questions, a study was conducted using two categories of analysis: the LR with its associated methods, and the SLR, along with their respective data collection techniques.
In this context of questions and reflections, the investigation into the qualitative approach to the IR, which resulted in this article, was conducted in two stages. In the first stage, a NR was performed on LRs and their various methods, with a focus on IR. The NR technique was selected as it is the most suitable method for initiating a study aimed at identifying the foundations, arguments, similarities, and differences of the research subject. In the second stage, a Research Protocol was developed, encompassing the initial phase of the IR process flow. This protocol formed an integral part of the literature review for a research project underway at the School of Education of the Federal University of Minas Gerais — UFMG (Alvarenga, 2019; 2021).
AN OVERVIEW OF THE LITERATURE REVIEW: CONCEPTUAL AND METHODOLOGICAL IMPLICATIONS
When considering knowledge as a collective construction of the scientific community, a researcher, similar to a public policy maker proposing a new research project or policy, initially reviews the existing scientific literature on their topic of interest. This stage provides the junior researcher with the opportunity to build upon the work of more experienced researchers, uncovering theories, concepts, methods, approaches, and contexts previously analyzed by their predecessors in a given field of knowledge.
This analogy suggests that the academic journey of a researcher is built upon a methodical LR. A thorough and insightful LR enables the researcher to gain a broader and more innovative perspective on new research possibilities within a specific scientific field. Despite its inherent limitations, research serves to highlight and build upon previous discoveries. Consequently, a LR can be defined as "a traditional method for ascertaining what is already known in a field of research" (Gough, 2007, p. 62).
According to Alves-Mazzotti (1992), an LR has two primary objectives: to build a contextualization for the research problem and to analyze the possibilities presented in the consulted literature to construct the theoretical framework for the study. Contextualizing the research object through an LR involves placing the researcher within the problem scenario throughout the study area and analyzing the theoretical framework. This process guides the researcher from defining the research problem to interpreting the results. Thus, LR is seen as a process of exploring both consolidated and emerging investigations, complementing and/or challenging existing scientific contributions related to the study object. It is essentially a search for secondary scientific research, involving the location and analysis of scientific publications in articles, books, theses, and printed or electronic dissertations on a given topic, followed by the publication of a synthesis of the conducted studies.
Proposing a research problem in LR requires researchers to immerse themselves in the process to assess the current state of knowledge in their area of interest. This involves comparing and contrasting theoretical-methodological approaches, evaluating the weight and reliability of research results, and identifying points of consensus, controversies, gaps, and areas needing clarification. Following this ideal research trajectory helps researchers to more precisely define their study object and to select appropriate theories, procedures, and instruments, or to avoid those that prove ineffective in the pursuit of the desired knowledge (Alves-Mazzotti, 1992).
Yin (2016, p. 79) asserts that LR aims to "bring together what is known about a given topic, potentially highlighting controversial or disparate lines of thought or even progress over time in the accumulated knowledge on a topic". This perspective requires the creation of a synthesis based on various bibliographic studies to develop a comprehensive understanding of a particular academic field. Consequently, LR becomes the foundational step in constructing scientific knowledge across all disciplines. Its significance is underscored by the presence of review journals in every disciplinary and thematic area of science. Through this process, new theories, approaches, and research methodologies emerge, and opportunities to identify gaps and avenues for future research in a specific field are revealed.
Given the vast amount of research and publications produced annually in the field of education (Vosgerau and Romanowski, 2014), LR can be a challenging task. Academics may be tempted to search for literature without clear criteria, potentially leading to a review lacking explicit procedures. In some cases, it can be difficult to determine what was reviewed and how (Gough, 2007). For an LR to yield results with empirical value, it is crucial to employ a specific scientific method that ensures its validity. According to Ingram et al. (2006), various methods are available for conducting an LR, and it is up to researchers to select the one that best suits their investigation.
As corroborated by various authors analyzed by Vosgerau and Romanowski (2014), LR is referred to by several different names, including bibliographic survey, LR, NR, bibliographic review, state of the art, bibliometric study, systematic review, IR, meta-analysis, meta-summarization, and synthesis of qualitative evidence. According to these authors, while these terminologies appear across different areas of knowledge, all referenced research addresses topics related to education (Vosgerau and Romanowski, 2014).
Considering the conceptual and methodological implications of the approaches highlighted by Vosgerau and Romanowski (2014), this study categorizes the types of reviews into two distinct groups: mapping reviews and evaluative and synthesizing reviews. Mapping reviews pertain to the initial stage of a bibliographic survey and aim to gather all secondary references on a given topic. They involve compiling studies from books, websites, journals, and videos, providing a broad overview of the investigated subject without employing detailed or specific criteria for source selection. This approach addresses the topic in a general manner, lacking methodological systematization and not specifying how the references were collected, whether manually or through specific software, such as EndNote3 and Mendeley,4 which facilitate the export of localized references to the researcher's devices.
Reviews that analyze and synthesize, in contrast to mapping reviews, involve establishing an investigative question, defining critical diagnosis and search strategies, and requiring transparency in the criteria for the inclusion and exclusion of studies. These reviews necessitate the use of primary data collected by the researcher, such as surveys, interviews, observations, reports, etc. (Davies, 2007). Based on these distinctions, LR grouping approach has been organized into two categories: mapping reviews and evaluation and synthesis reviews, as depicted in Figure 1.

Source: prepared by the authors based on the studies by Vosgerau and Romanowski (2014).
Figure 1 Organizational chart of literature review approaches.
In the LR process, the researcher faces the challenge of selecting the method that most effectively captures the results of research published in their area of study. While LR collection methods share certain similarities, each approach serves a distinct purpose, as each has different criteria for collection and analysis based on the research objectives (Rother, 2007). Whittemore and Knafl (2005) argue that the dissemination of various methodologies in LR research has led to the adoption of more systematic and rigorous methods.
Figure 2 illustrates the two types of research methodologies in LR: NR and SLR (Rother, 2007). NR is categorized under the mapping approach, while SLR falls under the evaluation and synthesis approach. The following subsections will detail the key aspects of each typology and highlight their main differences.

Source: prepared by the authors based on the studies by Rother (2007).
Figure 2 Organizational chart of types of literature review.
NARRATIVE LITERATURE REVIEW
NR, as a literature research method within the field of mapping reviews, is suited for mapping references on a given subject. It facilitates new analyses, descriptions, and arguments from a theoretical or contextual perspective based on published investigations. In this type of review, there is no requirement for rigorous documentation regarding the sources used, the methodology employed in searching for references, or the criteria for evaluating and selecting works.
In NR, researchers select works based on their theoretical perspective and the context of the topic addressed, creating a synthesis of research from magazine articles, printed and/or electronic newspapers, theses, dissertations, and books related to the study object. This selection is based on the researcher's interpretation and critical-personal analysis. Because the search criteria are not specified and publications are chosen randomly, NR results are often influenced by the researcher's personal bias, with critical analysis reflecting a subjective viewpoint. These factors contribute to the scientific validity of NR being more readily questioned (Rother, 2007).
Instead of using the term NR, which is more commonly used in the field of health (Vosgerau and Romanowski, 2014), Yin (2016) refers to it as "traditional research." The author indicates that, when reading a study for the first time to prepare an RN, researchers should follow these steps to capture information from the references researched: i. identify the study's topic, including the research problem and/or question; ii. identify the research methodology, including data collection techniques, the extent of data collection, and the research context; iii. identify the main results of the studies, including the specific date used to represent the results and the study's main conclusions; and, finally, iv. make their own comments on the study's strengths and weaknesses and provide complete bibliographic details for citing the study.
Yin (2016) advises that the information collected in NR should be recorded in tables and/or Excel spreadsheets. This can be facilitated using applications such as Rayyan — Qatar Computing Research Institute (QCRI)5 — and/or reference management software such as Mendeley, EndNote, or Zotero, among others.
To successfully conduct an NR, it is essential to establish the key pillars of LR empirics: the theme, the method, and the result. Developing conclusions based on these foundations is crucial. This pillar is represented in Figure 3.

Source: prepared by the authors based on the studies by Rother (2007) and Yin (2016).
Figure 3 Three pillars of empirical foundations in narrative review.
SYSTEMATIZED LITERATURE REVIEW
In the evaluation and synthesis approach, SLR serves as a methodical synthesis of research. This method supports evidence-based policies and practices by providing empirical backing for increasing interventions and scientific information, which are rapidly growing and influencing decision-making processes in education. SLR involves various types of systematized reviews that aggregate research evidence on a topic or issue, critically evaluating it in terms of methodology and conclusions, and determining the consistent and variable messages derived from that body of work (Davies, 2007).
Systematized bibliographic reviews involve synthesizing primary studies that have clearly defined objectives, materials, and methods, and were conducted following a transparent and reproducible methodology. This approach entails a series of pre-defined steps, with a specified methodology and standardized techniques that ensure reproducibility.
Based on studies by Whittemore and Knafl (2005), SLR can be conducted using various methods, including IR, systematic review, meta-analysis, and meta-summary, as illustrated in Figure 4.

Source: prepared by the authors based on the studies by Whittemore and Knafl (2005).
Figure 4 Organizational chart of methods for systematic literature reviews.
Galvão, Sawada, and Trevizan (2004) assert that the initial phase of conducting SLR, regardless of the method used, involves developing a protocol to ensure that the review is conducted with the same rigor as research. The basic components of this protocol include: i. the review question; ii. inclusion criteria; iii. search strategies and the definition of how research will be critically evaluated; iv. data collection; v. data synthesis; and vi. dissemination of results. Careful planning of the SLR is essential, and it is recommended that the protocol be reviewed by a qualified professional before starting the review. In this study, the IR Protocol was evaluated by the librarian of the Researcher Support Sector of the UFMG University Library.
NARRATIVE REVIEW X SYSTEMATIZED LITERATURE REVIEW
One of the key distinctions between SLR and NR is the scope of the search for potentially relevant primary studies. SLRs help avoid the generic readings often encountered in NRs by conducting comprehensive searches and critically analyzing all available research literature on the topic. They also differ in how they formulate research questions, their thorough research approach, their critical diagnostic strategies, the transparency of criteria for including or excluding primary studies, and the development of a detailed protocol (Davies, 2007).
NRs are recommended for mapping available scientific production and for (re)constructing networks of concepts that integrate knowledge from various sources to seek new insights into the research topic (Cordeiro et al., 2007). However, according to these authors and, based on studies by Segura-Muñoz et al. (2002), RN has a "descriptive-discursive" character, which leads to the non-presentation of "characteristics of reproducibility and repeatability, making it too empirical, obscure, and/or inconclusive in the opinion of some researchers" (Cordeiro et al., 2007, p. 05). The authors emphasize, however, that this "statement does not deny the importance of narrative reviews;" however, it opens windows to new possibilities and other RL methods, still little used.
With the introduction of SLR across various fields in recent decades, new methods for analysis and synthesis have been developed. The health sciences, in particular, have emerged as a leading area for the consolidation of systematized reviews. This has characterized SLRs within the scientific domain as having well-defined methods and high methodological rigor, distinguishing them from NR, which often serve as a general survey of available literature. Consequently, SLR methods have been employed to address the limitations and gaps left by NRs (Sampaio and Mancini, 2007).
INTEGRATIVE REVIEW: A THEORETICAL AND APPLICABILITY ANALYSIS
This article addresses only one of the four SLR methods — the IR. The theoretical framework for IR is primarily based on the studies of Cooper (1984), Ganong (1987), Broome (2006), and Whittemore and Knafl (2005). The applicability of IR is examined by comparing the references cited in this article with the practical application of the method in the Alvarenga's research project (2019). The method was selected for two main reasons: i. to facilitate the synthesis and analysis of existing scientific knowledge on the investigated topic: specifically, the Direct Money to School Program (Programa Dinheiro Direto na Escola — PDDE) as a tool for enhancing intergovernmental relations within the Brazilian federation through school management processes (Alvarenga and Oliveira, 2021); and ii. to present a theoretical model for understanding "democratic school management" based on the analysis of Intergovernmental Relations (IGR) induced by PDDE (Alvarenga, 2019).
IR is a specific method within SLR designed to summarize both empirical and theoretical literature to offer a more comprehensive understanding of a particular phenomenon (Broome, 2006). IR aims to integrate extensive information on a topic, regardless of whether the data is quantitative or qualitative. This method can be used to review existing theories, propose new concepts, and identify gaps in research.
The term "integrative" in IR reflects the method's focus on integrating diverse opinions, concepts, or ideas from the research sources utilized. The methodology for searching sources is clearly defined, and the material analyzed may include original studies, theoretical reviews, and case reports. In IR, there is a requirement for specificity regarding the study object, the collection and analysis of primary studies, the definition of search criteria, and the summary analysis of studies, along with the establishment of proven scientific evidence. The primary aim of IR is to develop a thorough understanding of a particular phenomenon based on existing research.
According to Sampaio and Mancini (2007), among the SLR methods, IR is the broadest. It has the advantage of allowing the simultaneous inclusion of qualitative, experimental, and quasi-experimental research, which provides a more comprehensive understanding of the topic of interest. This capability enables the combination of data from both theoretical and empirical literature. Consequently, IR can be utilized for various purposes, including defining concepts, reviewing theories, or conducting methodological analyses of the included studies on a given topic.
For Whittemore and Knafl (2005), the process of literature analysis using IR involves several key steps: formulating a research problem, cataloging literature, defining procedures for selecting literature, critically evaluating the data, analyzing the data, and presenting the results.
Based on the studies conducted, the process of preparing an IR is well-defined in the literature. However, different researchers may adopt specific methods for defining the stages of IR. In the study that resulted in this article, the approach of Whittemore and Knafl (2005) was chosen. According to their framework, the IR process should follow a sequence of six well-defined steps, which are illustrated in Figure 5 and will be detailed below.

Source: prepared by the authors based on the studies by Whittemore and Knafl (2005).
Figure 5 Functional chart of the Integrative Review.
Based on the six steps of IR, the process facilitates the systematic and orderly gathering, analyzing, and synthesizing of research results on a specific topic or question, thereby contributing to a deeper understanding of the investigated subject. Figure 5 graphically illustrates these six steps of the IR method and their fundamental functioning, which researchers must follow rigorously.
THEORETICAL ARCHITECTURE OF THE SIX STAGES OF INTEGRATIVE REVIEW
In this subsection, the six steps of IR are theoretically detailed. The specific aspects of implementing the IR will be addressed in the section titled "Protocol of an IR," which will be presented in the following subsection.
The first stage of the process involves identifying the topic and selecting the research question, which is crucial for guiding the construction of the IR protocol. The organization of this stage underpins the theoretical framework of the review and includes pre-established definitions from prior research or a research project. This stage begins with the clear and specific definition of a problem and the formulation of a research question, as it serves as the foundation for the entire research process (Mendes, Silveira and Galvão, 2008).
After formulating the research question, the next step involves defining the descriptors and/or keywords, referred to as ‘search terms,’ which will support the formulation of search strategies within the selected databases (Broome, 2006). The search strategy is a technique or set of rules designed to align a formulated question with the information stored in a database. Lopes (2002) emphasizes the importance of appropriately identifying the descriptive elements of an item and/or record within a database as a fundamental aspect of planning the search strategy. For the author, a critical first step involves examining the database documentation to identify the coding defined for each field of the information item. This ensures the selection of a search strategy that is coherent with the databases to be consulted by the researcher. Each step taken in this initial stage of the IR process should be documented and systematized within a specific document known as a Protocol.
In the second stage, inclusion and exclusion criteria are established. Following the completion of all steps in the first stage, researchers begin searching the databases to identify studies that will be included in the review. This stage is heavily influenced by the outcomes from the previous stage, as a broadly described problem will likely result in a diverse sample, requiring the researcher to apply more stringent analysis criteria. Often, the selection of articles starts broadly and then narrows as researchers revisit their initial question, as the search process in the literature is not always linear (Broome, 2006). Inclusion and exclusion criteria should be clearly and objectively identified in the study, though they may undergo reorganization during the article search process and during the preparation of the IR.
In the third stage, the focus is on highlighting pre-selected and selected studies. To identify these studies, the titles, abstracts, and keywords of all complete publications located through the search strategy are carefully read to assess their suitability against the study's inclusion criteria. When the title, abstract, and keywords are insufficient to determine selection, the entire article is reviewed. Scholars in SLR recommend that this evaluation process, conducted in the third stage of an IR, involves the analytical and discerning perspective of more than one reviewer.
In the fourth stage, the categorization of studies takes place. This stage is designed to summarize and document the information extracted from the scientific articles identified in the previous stages. The documentation must be prepared concisely and in a straightforward manner (Broome, 2006). The information collected from the articles should include details such as sample size, the number of subjects, methodology, measurement of variables, analysis methods, and the theories or underlying concepts utilized (Ganong, 1987).
In this process, to effectively extract information from articles, researchers must employ an instrument that allows for the separate analysis of each article (both in terms of methodology and research results), while also enabling the synthesis of articles, taking care to preserve their differences. This step is akin to data analysis in traditional scientific research. To analyze the information gathered from scientific articles, researchers should create analytical categories that facilitate the organization and summarization of each study. This categorization can be done descriptively, where researchers highlight the most relevant data for their study (Broome, 2006). Various methods can be used to categorize and analyze information, such as data synthesis, lists of factors demonstrating the effect of variables over the study period, and the selection or exclusion of studies, among others (Mendes, Silveira and Galvão, 2008). The process of creating categories can be approached in different ways. For instance, one method is listing variables by selecting valid studies closely related to the research topic, discarding those that do not align with the guiding question defined by researchers. It is essential for researchers to clearly explain the method used to analyze the data extracted from the scientific articles (Ganong, 1987).
In the fifth stage, results are analyzed and interpreted. This stage involves discussing the texts analyzed in IR. Researchers, guided by the findings, interpret the data and are thus able to identify existing gaps in knowledge and suggest guidelines for future research (Ganong, 1987; Mendes, Silveira and Galvão, 2008). To validate the study, it is crucial for researchers to clearly articulate the gaps identified in the literature and propose directions for future research that other researchers might follow in their scientific inquiries (Ursi and Galvão, 2005). Scholars in SLR recommend that reviews selected through this approach be subjected to the analytical and discerning evaluation of more than one reviewer.
In the sixth and final stage, the presentation of the review/synthesis of knowledge is prepared, aiming to enable the replication of the study. This stage involves producing an article or research report that meticulously describes all the phases covered by researchers and presents the main results obtained. According to Mendes, Silveira, and Galvão (2008, p. 763), this stage is of "extreme importance, as it produces an impact due to the accumulation of existing knowledge on the researched topic." The review/synthesis of knowledge is then presented, detailing the research process as thoroughly as primary source research. The presentation of data should provide readers with comprehensive information about the studies reviewed, focusing not only on the most significant findings but also including the procedures used to conduct the review, aspects related to the topic investigated, and the design of the studies included.
Whittemore and Knafl (2005) emphasize that the methodological rigor and systematization of IR are crucial to its potential for contributing to scientific knowledge. However, they also caution researchers about the importance of careful execution when conducting an IR. Without explicit and systematic methods, the likelihood of errors increases significantly. Such errors can occur at any stage of the review process. For instance, the bibliographic research phase might be incomplete if essential primary sources are overlooked, or data from these sources could be inaccurately extracted or misinterpreted. Therefore, it is essential that the IR is conducted with rigor and transparency to ensure that the information provided allows readers to assess the relevance and validity of the procedures used in the review process.
PROTOCOL PROTOTYPE: INTEGRATIVE REVIEW FLOW OF RESEARCH INTO EDUCATIONAL PUBLIC POLICIES
This subsection presents the IR Protocol from the literature review of the research project by Alvarenga (2019) as a potential application of IR in studies within the field of educational policies. The Protocol details all items of each stage of the IR along with their contents. It begins with a section titled "Identification", which includes information such as: the title of the research that underlies the review; research objectives, providing readers with sufficient information to understand the research topic and the type of study; and the methodology of the IR.
It is crucial to define the stages of the methodology, which can vary among different method scholars. In this instance, the six steps outlined by Whittemore and Knafl (2005) were adopted. Additionally, the roles of the researchers involved in the review process were clearly defined. This includes assigning responsibilities for planning the research, writing the Protocol, and conducting the systematic search and selection of studies.
In the Protocol, the following items should address the methodological questions regarding the IR process: Who reviewed the articles? Was there more than one reviewer involved? Were the articles evaluated independently by each reviewer? In what way?
In the Protocol, two reviewers from the Research Group on Policy and Administration of Educational Systems (Grupo de Pesquisa em Política e Administração de Sistemas Educacionais — GruPase/FaE/UFMG) were invited to analyze the references tracked by applying the search strategy in the databases. During the initial stage of the selection process, where the title, abstract, and keywords were evaluated, each study was assessed independently by each reviewer. For the subsequent evaluation of the full text, only the author of the research project performed this task (Alvarenga, 2019). Chart 1 provides a summary of the essential elements that should constitute an IR Protocol, illustrating the method's applicability.
Chart 1 Integrative Review Protocol.
| IDENTIFICATION |
|---|
|
| 1st STAGE |
| Theme: Public funding of educational management. |
| Topic: Financial School Management. |
| Study Object: PDDE. |
| Research Subjects: state public schools located in the state of Minas Gerais. |
| Research Context: school management based on the Intergovernmental Relations (IGR) induced by PDDE. |
| Research Temporal Scope: January/1995 to January/2022. |
| Research project hypothesis: PDDE, by inducing relations between the Ministry of Education (MEC), the National Fund for Education Development (Fundo Nacional de Desenvolvimento da Educação — FNDE), state and/or municipal Education Secretariats, Executive Units (UEx), and public schools, reorganizes the democratic school management prescribed in the National Education Guidelines and Bases Law of 1996 (Lei das Diretrizes e Bases da Educação Nacional de 1996 [Brasil, 1996)]. |
| Research Problems to be Addressed by the IR: Did IGR induced by PDDE in school management redefine the model of democratic school management as defined in LDB/96? How and what are the center-school relations induced by PDDE in school management? What do empirical studies conducted from January 2000 to January 2022 reveal about the center-school relations induced by PDDE in school management? What is known and/or how little is known about a theoretical model for understanding "democratic school management" based on the analysis of IGR induced by PDDE? |
| Analysis categories: School Management; PDDE; Intergovernmental Relations. |
Descriptors defined from analysis categories:
|
| Guiding Question: What is the definition of "democratic school management" in the researched literature? |
| 2nd STAGE |
Search Strategies: The sampling for selection was determined using the following inclusion and exclusion criteria:
|
| Search Terms: (Programa Dinheiro Direto na Escola OR Direct Money in School Program) AND (Gestão Escolar OR Gestão Escolar Democrática OR Relações Intergovernamentais OR Financiamento PDDE). |
| Use of Boolean Operators (specific strategies): use of AND (limits the search, includes terms); OR (expands the search to include at least one of the indicated terms), being used for different terms with the same meaning — e.g., Programa Dinheiro Direto na Escola or PDDE; NOT (exclusion, limits the search, excludes articles containing the indicated term) — in this research, the term health, which frequently uses the acronym PDDE. |
| Databases: http://bdtd.ibict.br/vufind/ http://www-periodicos-capes-gov-br.ezl.periodicos.capes.gov.br/index.p |
| Justification for database selection: |
| The databases were selected based on the following aspects, considered sufficient for conducting this IR. |
| The BDTD, developed by IBICT*, includes most repositories of theses and dissertations generated from research at higher education institutions nationwide. Another factor was the operability of the search engine, with a more efficient search interface for retrieving and filtering the researched topics compared to the Theses and Dissertations Catalog of the Capes. |
| For reviewing articles, the Portal of Periodicals of Capes** is the largest virtual library of scientific information directed at the national academic community, consisting of paid research databases subscribed to by Capes and open-access databases. The content is available for free to the academic community of educational and research institutions in Brazil. |
| On BDTD of IBICT: |
| 1. The "advanced search" was used for the defined terms, in the appropriate fields. |
| 2. The period from 1995 to 2022 was defined. |
| 3. Theses and dissertations were saved in IR folders on the computer, organized and identified separately for dissertations (master's) and theses (doctorate), according to the search terms by subject. The files were then exported to Mendeley, as the BDTD/IBICT platform did not allow for direct export of files to Rayyan. |
| On the Capes Portal: |
| 1. Personalized services offered by the CAFE tool, from the Federated Academic Community, were used for access and identification by the academic community. The tool enables remote access (from computers outside the institution and even abroad) to subscribed (paid) content from the Portal of Periodicals. Access is based on user identification (students, staff, professors, researchers, and, specifically in the case of UFMG, also alumni). The login and password from Minha UFMG are used for identification. The Capes Portal also offers the "My Space" tool, which allows saving searches, favoriting articles, and using the alert service to receive notifications of new indexed articles with chosen terms. UFMG's Library System includes the Research Support Sector (https://www.bu.ufmg.br/bu_atual/o-sistema-de-bibliotecas/apoio-ao-pesquisador/), offering individual and group training, both in-person and remote, through its YouTube channel (https://www.youtube.com/c/Suporte%C3%A0PesquisaUFMG), where presentations by guests on databases and research tools are recorded. The Sector mediates issues and suggestions related to the use of the Portal with editors and Capes. |
| 2. Strategy with Search Strategies on the Journal Portal: In the advanced search, the term Programa Dinheiro Direto na Escola was used (in any field, meaning search in title, keywords); the (exact) match was selected in the second field to ensure the integrity of the compound term, excluding the possibility of searching isolated words. The Boolean OR was added to search for the acronym PDDE using the (exact) match to ensure accuracy. The NOT operator was also added to exclude works related to health, where the acronym appeared, as demonstrated below: |
|
| Date Filter: Selected "last 22 years;" language filter "English, Portuguese, Spanish"; start date: 01/01/2000; end date: 01/31/2022. Review Filter: selected "show only peer-reviewed journals." Material Type Filter: Articles. All filters were used as demonstrated below: |
|
| 3 References were exported to EndNote and organized according to search terms, and then attached to Rayyan, as shown in the following screenshot: |
|
| Organization of collected material from databases: |
| 3rd STAGE |
Identification of Pre-Selected and Selected Studies
|
| 4th STAGE |
Organization of Selected Studies
|
| 5th STAGE |
| Analysis and Interpretation of Results |
| 6th STAGE |
| Writing the Review/Synthesis of Knowledge |
Source: prepared by the authors based on Alvarenga (2019; 2021) and Alvarenga and Oliveira (2021).
*IBICT developed and coordinates BDTD, which electronically integrates the thesis and dissertation information systems of all higher education and research institutions in Brazil, providing greater visibility to national scientific production. Source: https://bdtd.ibict.br. Access on: Mar. 20, 2021.
**The Capes Periodicals Portal aims to strengthen graduate programs in Brazil through the democratization of online access to scientific information. The Portal provides access to a variety of electronic content through its website, including texts available in over 45,000 national and international periodicals; various databases compiling academic and scientific papers, patents, theses, and dissertations, among other types of materials, covering all areas of knowledge. The portal can be accessed through computers connected to the internet within institutions or remotely via a user ID and password. The latter method is known as CAFE. Source: https://www-periodicos-capes-gov-br. Access on: Mar. 20, 2021.
In short, the flow of the IR process is presented in Figure 6. The IR procedure is divided into six distinct, but complementary, stages, namely: i. preparation of the Research Protocol, which comprises the systematization of the planned activities in the first and second stages of IR; ii. literature management — this involves the management of selected literature, which corresponds to the actions of the third stage; iii. data collection — moment of organizing the researched bibliography, which refers to the questions defined in the fourth stage; iv. data analysis — stage where examination, interpretation and discussion of findings in the literature are carried out, actions that relate to the activities of the fifth stage; v. review synthesis, corresponding to the preparation of reports, a stage that harmonizes with the activities of the sixth stage; vi. writing and publishing of results, when articles are prepared that describe the review in detail.
FINAL CONSIDERATIONS
IR has the potential to promote review studies in different areas of knowledge while maintaining the methodological rigor of systematic reviews. This method emerged as an alternative to rigorously review and combine studies with different methodologies and integrate scientific results.
Based on the analytical study presented in this article, which aimed to highlight the possibilities of using IR in the field of education, it is concluded that the integrative systematic review in the area of educational public policies can assist researchers, academics, policymakers, and professionals in making well-founded decisions regarding public educational policies. The method facilitates the incorporation of results on the effectiveness or otherwise of the methodology used in the investigation, as well as scientific analyses in the field of empirics of a given public policy. Therefore, IR constitutes a valuable method for conducting an LR in the field of educational public policies, as it enables the combination of data from empirical and theoretical literature, which can be directed toward defining concepts, identifying gaps in the areas of study, reviewing theories, and analyzing methodological aspects of studies on a given topic. The integration of research with various methods in the integrative review broadens the possibilities for analyzing the literature.
By presenting a specific protocol in research conducted in the field of public educational policies, this article demonstrates the applicability of this methodology within this subarea. IR can thus be regarded as an adaptable technique for conducting LRs in this field. While it is acknowledged that combining data from various qualitative and quantitative research designs is complex and challenging, employing IR, through the incorporation of a rigorous system, leads to a reduction in biases and errors in review work.
Once established, IR can be confirmed as a valid tool for LRs within the current context of public educational policies in Brazil. The diversity of bibliographical references available for inclusion in an IR sample, combined with the method's multiple purposes, results in a comprehensive overview of concepts, theories, methodologies, and issues related to public policies in the educational sector.
Despite the possibilities for applying IR in the educational dimension highlighted in this manuscript, and recognizing the gaps identified in studies that include IR as an SLR method, it is necessary to intensify efforts in developing research with methodological designs of this method, particularly in the empirics of public educational policies.
This study is expected to contribute to the academic community by promoting greater methodological rigor in education research. It aims to provide an overview of how to conduct IR in practice and serve as a basis for new research on the topic.
1In this article, we propose using the term systematized literature reviews (SLR) to encompass the methods of integrative review, systematic review, meta-analysis, and summary. This definition was chosen instead of the term "systematic review" proposed by Whittemore and Knafl (2005) to avoid ambiguous understanding with the "systematic review" method.
2Data that supports and/or refutes a statement. There are several types of evidence available to researchers, academics, and professionals to support ideas and propositions that emerge as part of research or work: observation of documents, listening to others’ words, reasoning or reflection, and research in general, regardless of type (Thomas, 2007).
3Bibliographic reference management software developed by Clarivate Analytics, available for free on the Web of Science platform. Access and registration can be done through the Coordination for the Improvement of Higher Education Personnel (Capes) Periodicals Portal at http://www.periodicos.capes.gov.br. Access on: Apr. 10, 2021.
4Free software provided to academics by a company based in London, UK. The software is primarily known for its reference manager, used to manage, share, read, annotate, and edit scientific articles, theses, and dissertations, as well as to create a research network for managing online LR analyses, discovering research trends, and connecting with other researchers with similar interests in a study area. Source: www.mendeley.com. Access on: May 15, 2021.
5Free web application funded by the Qatar Computing Research Institute, a non-profit organization from Qatar, primarily used to assist in SLR. It offers a variety of features, including creating a review project, inviting collaborators, exploring citations in different formats, exporting, importing, labeling and filtering citations, categorizing into included, excluded, and "in doubt" references, blinding between reviewers, and automatic identification of potential duplicates. Source: https://www.rayyan.ai. Access on: June 21, 2021.
Funding:Intergovernmental relations in the management of public schools in the State of Minas Gerais: an analysis through the PDDE (2007-2020)", which had financial support from the Fundação de Amparo à Pesquisa do Estado de Minas Gerais (Fapemig), n. APQ 01372-21.
REFERENCES
ALVARENGA, Eldaronice Queiroz de. Projeto de Pesquisa Relações Intergovernamentais na Gestão das Escolas Públicas do Estado de Minas Gerais: uma análise por meio do Programa Dinheiro Direto na Escola - PDDE (2007-2020). Belo Horizonte: UFMG, 2019. [ Links ]
ALVARENGA, Eldaronice Queiroz de. Base de dados da revisão de literatura da pesquisa Relações Intergovernamentais na Gestão das Escolas Públicas do Estado de Minas Gerais: uma análise por meio do Programa Dinheiro Direto na Escola - PDDE (2007-2020). Belo Horizonte: UFMG, 2021. [ Links ]
ALVARENGA, Eldaronice Queiroz de; OLIVEIRA, Rosimar de Fátima. PDDE: Mecanismo Indutor de Relações Intergovernamentais na Gestão das Escolas Públicas Brasileiras. Seminários Regionais da ANPAE [Recurso Eletrônico] Organizadores: Gilda Cardoso Araujo, Itamar Mendes da Silva, Lilian Marques Freguete, Lorrainy Ferrari e Rosenery Pimentel do Nascimento. Local: Vitória, ES: nº 06, 2021. Tema Central: "A educação na pandemia: dilemas da democracia e o direito à Educação". ISSN 2595-5705. Disponível em: https://www.seminariosregionaisanpae.net.br/index.htm. Acesso em: 08 fev. 2022. [ Links ]
ALVES-MAZZOTTI, Alda Judith. A "revisão bibliográfica" em teses e dissertações: meus tipos inesquecíveis. Cadernos de Pesquisa, São Paulo, n. 81, p. 53-60, maio 1992. [ Links ]
BRASIL. Lei n. 9.394, de 20 de dezembro de 1996. Dispõe sobre as Diretrizes e Bases da Educação Nacional. Diário Oficial da União, Brasília, 23 dez. 1996. Disponível em: https://www.planalto.gov.br/ccivil_03/leis/l9394.htm. Acesso em: 06 jul. 2010. [ Links ]
BROOME, Marion E. Integrative literature reviews for the development of concepts. In: RODGERS, Beth L.; KNAFL, Kathleen A. Concept development in nursing: foundations, techniques, and applications. 2. ed. Philadelphia, PE: W. B. SaundersCompany, 2006. p. 231-250. [ Links ]
COOPER, Harris M. The integrative research review: a systematic approach. Beverly Hills, CA: Sage Pulications, 1984. [ Links ]
CORDEIRO, Alexander Magno; OLIVEIRA, Glória Maria de; RENTERÍA, Juan Miguel; GUIMARÃES, Carlos Alberto. Revisão sistemática: uma revisão narrativa. Grupo de Estudo de Revisão Sistemática do Rio de Janeiro (GERS-Rio). Comunicação Científica. Revista do Colégio Brasileiro de Cirurgiões, Rio de Janeiro, v. 34, n. 6, p. 428-431, nov.-dez. 2007. Disponível em: https://www.scielo.br/j/rcbc/a/CC6NRNtP3dKLgLPwcgmV6Gf/?lang=pt. Acesso em: 10 dez. 2021. [ Links ]
DAVIES, Phillip. Revisões sistemáticas e a Campbell Collaboration. In: THOMAS, Gary; PRING, Richard. Educação baseada em evidências: atualização dos achados científicos para a qualificação da prática pedagógica. Porto Alegre: Artmed, 2007. p. 31-43. [ Links ]
GALVÃO, Cristina Maria; SAWADA, Namie Okino; TREVIZAN, Maria Auxiliadora. Revisão sistemática: recurso que proporciona a incorporação das evidências na prática da enfermagem. Revista Latino-Americana de Enfermagem, Ribeirão Preto, v. 12, n. 3, p. 549-556, maio-jun. 2004. https://doi.org/10.1590/S0104-11692004000300014 [ Links ]
GANONG, Lawrence H. Integrative reviews of nursing research. Research in Nursing & Health, Hoboken, v. 10, n. 1, p. 1-11, mar. 1987. https://doi.org/10.1002/nur.4770100103 [ Links ]
GATTI, Bernadete A. Grupo focal na pesquisa em Ciências Sociais e Humanas. Brasília: Líber Livros Editora, 2012. [ Links ]
GOUGH, David. Síntese sistemática de pesquisa. In: THOMAS, Gary; PRING, Richard. Educação baseada em evidências: atualização dos achados científicos para a qualificação da prática pedagógica. Porto Alegre: Artmed, 2007. p. 58-76. [ Links ]
INGRAM, Laura; HUSSEY, James; TIGANI, Michelle; HEMMELGARN, Mary. Writing A Literature Review and Using a Synthesis Matrix. Tutorial services. [s.l: s.n.], 2006. Disponível em: https://mville.libguides.com/ld.php?content_id=43140779&_x_tr_sl=en&_x_tr_tl=pt&_x_tr_hl=pt-BR&_x_tr_pto=sc. Acesso em: 10 dez. 2021. [ Links ]
LOPES, Ilza Leite. Estratégia de busca na recuperação da informação: revisão da literatura. Ciências da Informação, Brasília, v. 31, n. 2, p. 60-71, maio/ago. 2002. Disponível em: https://www.scielo.br/j/ci/a/VPbDtPgqnFs5cm8GSLhtsyM/?format=pdf&lang=pt. Acesso em: 10 dez. 2021. [ Links ]
MENDES, Karina Dal Sasso; SILVEIRA, Renata Cristina de Campos Pereira; GALVÃO, Cristina Maria. Revisão Integrativa: Método de Pesquisa para a Incorporação de Evidências na Saúde e na Enfermagem. Texto Contexto Enfermagem, Florianópolis, v. 17, n. 4, p. 758-64, out.-dez. 2008. Disponível em: https://www.scielo.br/j/tce/a/XzFkq6tjWs4wHNqNjKJLkXQ/. Acesso em: 10 dez. 2021. [ Links ]
ROTHER, Edna Terezinha. Revisão sistemática x revisão narrativa. Acta Paulista de Enfermagem, São Paulo, v. 20, n. 2, p. v-vi, jun. 2007. Disponível em: https://www.scielo.br/scielo.php?pid=S0103-21002007000200001&script=sci_arttext. Acesso em: 11 out. 2019. [ Links ]
SAMPAIO, Rosana Ferreira; MANCINI, Marisa Cotta. Estudos de revisão sistemática: um guia para síntese criteriosa da evidência científica. Revista Brasileira de Fisioterapia, São Carlos, v. 11, n. 1, p. 83-89, jan.-fev. 2007. Disponível em: https://www.scielo.br/j/rbfis/a/79nG9Vk3syHhnSgY7VsB6jG/?format=pdf&lang=pt. Acesso em: 10 dez. 2021. [ Links ]
SEGURA-MUÑOZ, Susana Inés; TAKAYANAGUI, Angela Maria Magosso; SANTOS, Claúdia Benedita dos; SANCHEZ-SWEATMAN, Otto. Revisão sistemática de literatura e metanálise: noções básicas sobre seu desenho, interpretação e aplicação na área da saúde. In: SIMPÓSIO BRASILEIRO DE COMUNICAÇÃO EM ENFERMAGEM, 8., 2002, Ribeirão Preto. Anais… Ribeirão Preto: SIBRACEN, Escola de Enfermagem de Ribeirão Preto da USP. Disponível em: http://www.proceedings.scielo.br/pdf/sibracen/n8v2/v2a074.pdf. Acesso em: 10 dez. 2021. [ Links ]
THOMAS, Gary. Introdução: evidências e prática. In: THOMAS, Gary; PRING, Richard. Educação baseada em evidências: atualização dos achados científicos para a qualificação da prática pedagógica. Porto Alegre: Artmed, 2007. p. 58-76. [ Links ]
URSI, Elisabeth Silva; GALVÃO, Cristina Maria. Prevenção de lesões de pele no perioperatório: revisão integrativa da literatura. Revista Latino-Americana de Enfermagem, São Paulo, v. 14, n. 1, p. 124-131, jan.-fev. 2006. Disponível em: https://www.scielo.br/j/rlae/a/7hS3VgZvTs49LNX9dd85VVb/?lang=pt&format=pdf. Acesso em: 10 dez. 2021. [ Links ]
VOSGERAU, Dilmeire Sant’Anna Ramos; ROMANOWSKI, Joana Paulin. Estudos de revisão: implicações conceituais e metodológicas. Revista Diálogo Educacional, Curitiba, v. 14, n. 41, p. 165-189, jan.-abr. 2014. Disponível em: https://periodicos.pucpr.br/dialogoeducacional/article/view/2317. Acesso em: 10 dez. 2021. [ Links ]
WHITTEMORE, Robin; KNAFL, Kathleen. The integrative review: updated methodology. Journal of Advanced Nursing, Portland, v. 52, n. 5, p. 546-553, 2005. Disponível em: https://onlinelibrary.wiley.com/doi/full/10.1111/j.1365-2648.2005.03621.x. Acesso em: 11 out. 2019. [ Links ]
YIN, Robert K. Pesquisa qualitativa do início ao fim. Tradução: Daniel Bueno. Revisão técnica: Dirceu da Silva. Porto Alegre: Penso, 2016. [ Links ]
Received: March 09, 2023; Accepted: August 22, 2023










texto em 






