SciELO - Scientific Electronic Library Online

 
vol.47Práctica pedagógica, metacognición y comprensión lectora en lengua extranjera: una revisión sistemática desde un enfoque deductivoVariáveis que influenciaram o IDEB do ensino fundamental das redes públicas municipais paulistas em 2017 índice de autoresíndice de assuntospesquisa de artigos
Home Pagelista alfabética de periódicos  

Serviços Personalizados

Journal

Artigo

Compartilhar


Educação e Pesquisa

versão impressa ISSN 1517-9702versão On-line ISSN 1678-4634

Educ. Pesqui. vol.47  São Paulo  2021  Epub 20-Set-2021

https://doi.org/10.1590/s1678-4634202147243789 

ARTICLES

Mixed methods approach in the theses of a postgraduate program in education: analysis in the light of Creswell1 *

Luciana Rodrigues Leite2 
http://orcid.org/0000-0003-1915-6462

Ana Paula dos Santos Reinaldo Verde3 
http://orcid.org/0000-0002-8138-7280

Francisco das Chagas Rodrigues de Oliveira4 
http://orcid.org/0000-0002-3478-1058

João Batista Carvalho Nunes5 
http://orcid.org/0000-0002-1270-0026

2- Universidade Estadual Vale do Acaraú, Sobral, Ceará, Brasil. Contato: luciana_leite@uvanet.br.

3- Secretaria de Educação do Estado do Maranhão, São Luís, Maranhão. Contato: napaulareinaldo@gmail.com.

4- Rede Municipal de Boa Viagem, Ceará e Rede Municipal de Monsenhor Tabosa, Ceará, Brasil. Contato: chicoverde.ufjf@hotmail.com.

5- Universidade Estadual do Ceará, Fortaleza, Ceará, Brasil. Contato: joao.nunes@uece.br.


Abstract

Mixed methods research embraces the association of procedures to collect and analyze data as well as the combination of quantitative and qualitative techniques. In the past years there has been an increase in the use of mixed methods, especially in the international literature. This article focuses on the use of mixed methods to education research in Brazil. The following question guides the investigation: “to what extent the mixed methods approach is present in Brazilian Postgraduate Programs in Education whose academic performance matches the international standards of excellence?”. A documentary research was conducted to map and analyze the theses of a postgraduate program that reached the highest score according to Coordination for the Improvement of Higher Education Personnel (CAPES) from 2013 to 2018. The studies of John Creswell provided our theoretical basis and the analysis criteria for the selected theses. The search conducted on the Sucupira Platform found that 11 of 104 theses (10.6%) from the appointed Postgraduate Program in Education make use of mixed methods. Thus, it was verified, to greater and lesser degrees, the perception of the potential use of mixed methods in the theses, which improves over time as more recent theses present clearer justifications for integrating qualitative and quantitative techniques. In addition, both theoretical and methodological developments were verified, however, researchers in the Postgraduate Program have yet to employ more up to date references regarding mixed methods research.

Key words: Mixed methods; Education research; Post-graduation in education; Theses

Resumo

A pesquisa de métodos mistos engloba a associação entre procedimentos de coleta, análise e combinação de técnicas quantitativas e qualitativas. Nos últimos anos ocorreu um aumento vertiginoso no uso da abordagem mista, sobretudo na literatura internacional. Este ensaio incide sobre o uso do método misto na pesquisa em educação, no Brasil, com fundamento na seguinte questão norteadora: “em que medida a abordagem mista de pesquisa está inserida, no Brasil, em programas de pós-graduação da área de educação (PPGEs), considerados de desempenho equivalente a padrões internacionais de excelência?”. Desenvolveu-se, portanto, pesquisa documental, na qual foram mapeadas e analisadas teses de um programa de pós-graduação, nota sete, de acordo com avaliação da Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), no período de 2013 a 2018. Os estudos de John Creswell forneceram a base teórica e os critérios de análise das teses selecionadas. As buscas foram realizadas na Plataforma Sucupira e, dentre as 104 teses identificadas como pertencentes ao PPGE em foco, no referido período, 11 (10,6%) foram identificadas como concernentes ao contexto das investigações do tipo mistas. Considera-se que há, em maior e menor grau, a percepção do uso potencial de um projeto de método misto nas teses analisadas, com evolução temporal na melhoria dessa percepção, pois teses mais recentes apontam justificativas mais claras para a integração quali-quantitativa. Identificou-se também evolução teórica e metodológica; todavia, há necessidade de os pesquisadores do referido programa buscarem referências mais atuais no tocante aos estudos do tipo misto.

Palavras-Chave: Abordagem mista; Pesquisa em educação; Pós-graduação em educação; Teses

Introduction

This article maps and analyzes the application of mixed methods research in theses of a Brazilian Postgraduate Program in Education, score seven according to the standards used by the Coordination for the Improvement of Higher Education Personnel (CAPES). This work relied on the assumptions posed by Creswell (2012) who conceives mixed methods research as the association of procedures to collect and analyze data as well as the combination of quantitative and qualitative techniques. Given their interdisciplinary nature, mixed methods research is seen as an improvement for scientific work since the employment of the best features of each approach leads to investigations of optimal quality. Creswell’s perspective also informed the analysis of our sample, since we relied on his outline of the characteristics to elaborate and to validate mixed methods investigation to develop the present article.

John W. Creswell is a professor of Educational Psychology in the Post-Graduation in the Qualitative and Mixed Methods Research (QMMR) in Education at University of Nebraska-Lincoln. Choosing Creswell’s work as theoretical foundation of this investigation is justified on his renown in this field of expertise with numerous publications, which have been translated to many languages, but, mostly, due to his importance for mixed methods research and his intense work to popularize this ordered perspective in the pursuit of knowledge.

The awareness of the sudden increase in publications that employed mixed methods research, particularly in articles, originated this investigation. Galvão, Pluye and Ricarte (2018) point that such an increase is even more noticeable in foreign publications, which can indicate a path to be followed by Brazilian and Latin-American researchers. Therefore, the present article poses the following question “to what extent the mixed methods approach is present in Brazilian Postgraduate Programs in Education whose academic performance matches the international standards of excellence?”.

Answering this question demanded more specific ones be made, such as: a) how frequent are mixed methods used in theses written for a Postgraduate Program in Education score seven on the CAPES standard?; b) what assumptions support the method outlined for these investigations?; c) to what extent these theses follow Creswell’s methodological guidelines for mixed methods investigations?

The sample of this investigation was limited to one Postgraduate Program that scored seven on the CAPES standard, which indicates the excellency of its theses. This highlights the need to catalog and to analyze these publications to strengthen Postgraduate Programs in Education in Brazil and, more importantly, to contribute to the rigor of scientific work in this field.

Indeed, to preserve the scientific and methodological rigor of the present study, a temporal section from 2013 to 2018 was adopted. The search conducted on the Sucupira Platform found that 11 of 104 theses (10.6%) from the appointed Postgraduate Program in Education make use of mixed methods. These were analyzed according to the theoretical propositions of John Creswell given the criteria to be described in the methodology section of this work.

The present article is organized in four sections. In the first, the theoretical propositions upon which this work is based are presented through a short history of the mixed methods research, its definitions, characteristics and challenges to investigators. Next, the methodology adopted in this investigation is presented to the following presentation and analysis of theses that employed mixed methods. The fourth and last section gather the contributions of this research to its area of expertise as well as the conclusions made from the analysis of stricto sensu academic writings.

Theoretical assumptions of mixed methods research

A careful search for publications in both national and international journals of several areas of partially ordered knowledge reveals that studies that mix qualitative and quantitative research methods are becoming more common. The rise of mixed methods research has been pointed out by several authors like Cresswell (2013) and Ivankova (2015), who comment on the sudden popularity of that type of investigation.

Hence, the relevance of introducing some features related to the evolution of the concept of mixed methods. Creswell and Clark (2013) provide an interesting overview where they highlight that the growing interest in this approach by several researchers has given rise to several definitions for mixed methods that incorporate various elements of methods, research processes, philosophy, and research design.

At first, the mixed methods research proposed by Creswell and Clark (2013) was characterized by the mixing of both methods. Given the authors’ background in the field of evaluation, in 1989, they proposed an early definition disentangled from philosophical conceptions to support their claim. Ten years later, Tashakkori and Teddlie (1998) propose a broader definition that characterized mixed methods research as a methodological orientation in which two methods (qualitative and quantitative) are mixed in all phases of the research process from data collection to result analysis and even its philosophical positions.

Over time and the popularization of mixed methods investigations, definitions soon became more diverse and common. Some of these definitions would concern the development of mixed investigations, such as the one by Tashakkori and Creswell (2007, p. 4), who conceive it “[…] as research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or a program of inquiry”.

Given the existence of multiple viewpoints on the matter, Creswell and Clark (2013) formulate a different definition to cover all essential characteristics of the mixed methods research. This definition combines methods, a philosophy and the project design in addition to highlight the key components for designing and conducting a mixed methods research. According to the authors,

[...] the researcher collects and analyzes persuasively and rigorously both qualitative and quantitative data (based on research questions); mixes (or integrates or links) the two forms of data concurrently by combining them (or merging them), sequentially by having one build on the other, or embedding one within the other; gives priority to one or to both forms of data (in terms of what the research emphasizes); uses these procedures in a single study or in multiple phases of a program of study; frames these procedures within philosophical worldviews and theoretical lenses; and combines the procedures into specific research designs that direct the plan for conducting the study. (CRESWELL; CLARK, 2013, p. 22).

To confirm Creswell and Clark’s perspective, Sampieri, Collado and Lúcio (2013, p. 557) highlight that a “solid mixed study begins with the formulation of a categorical problem and demands clearly the use and the integration of quantitative and qualitative approaches […]”. These, in turn, are integrated from the beginning of the problem, passing through the sample, data collection, data analysis procedures and the interpretation of results. As a matter of fact, Creswell (2010) points out four key aspects to be considered by the researcher when planning a mixed methods study: distribution of the research timing; weight attribution; data combining procedure; and theorization, as shown in Chart 1. To facilitate the description of these strategies for mixed methods, a notation system has been developed, with researchers using shorthand labels and symbols to communicate their procedures. Cresswell and Clark (2013) suggest the use of a plus (+) to indicate methods that occur at the same time and an arrow (→) to indicate methods that occur in a sequence; “qual” and “quan” represent qualitative and quantitative, respectively, and uppercase letters indicate the prioritized methods. Therefore, the notation QUANT/qual means the priority of the quantitative method over the qualitative.

Chart 1 – Aspects to be considered in designing a mixed methods research 

DISTRIBUTION OF THE RESEARCH TIMING WEIGHT ATTRIBUTION COMBINING PROCEDURE THEORIZATION
The quantitative and qualitative data is collected either at the same time (concurrently) or in phases (sequentially).
  • It refers to the weight or priority given to the quantitative and qualitative approaches. The weight may be equal or emphasize one or the other.

  • (QUAL-QUAN; QUAL-quan; QUAN-qual)

  • It refers to the phases in which qualitative and quantitative data are combined: collection, analysis, interpretation or all of the phases.

  • Connected data – qualitative and quantitative data are connected during the research.

  • Integrated data – qualitative and quantitative data are fused and support one another mutually.

  • Embedded data – secondary data support a primary database.

Whether a larger theoretical perspective that guides the entire research project, which can be explicit or implicit.

Source: Adapted from Creswell (2010).

According to Creswell (2010), these four elements help the designing of strategies for mixed methods research. One strategy, according to Yin (2005), defines how data is collected and analyzed. One particular study analyzed by Creswell et al. (2003) point to the existence of six main strategies that researchers can adopt to design their research (but these do not exhaust all possibilities). Sequential mixed studies are divided into: sequential explanatory strategy; sequential exploratory strategy; sequential transformative strategy; whereas concurrent studies are divided into concurrent triangulation strategy; concurrent embedded strategy; concurrent transformative strategy, following the characteristics see in Chart 2.

Chart 2 – Mixed methods research strategies 

STRATEGY WEIGHT COMBINATION THEORIZATION DEFINITION
Sequential Explanatory Quantitative Connected Implicit The collection and analysis of quantitative data is carried out in one of the first stages of the research, followed by qualitative data collection and analysis. This second phase is developed from results of the first quantitative phase.
Sequential Exploratory Qualitative Connected Implicit The collection and analysis of qualitative data is carried out in one of the first stages of the research, followed by quantitative data collection and analysis. This second phase is developed from results of the first qualitative phase.
Sequential Transformative Qualitative or Quantitative Connected Explicit The initial phase is either qualitative or quantitative; it is distinguished from the other strategies due to a theoretical perspective that guides the investigation.
Concurrent Triangulation Qualitative and Quantitative (equal) Integrated Explicit or implicit The procedures for collection of qualitative and quantitative data are concurrent (at the same time) and followed by the comparison of databases.
Concurrent Embedded Qualitative or Quantitative Embedded Implicit The procedures for collection of qualitative and quantitative data are concurrent (at the same time) but a primary method (either qualitative or quantitative) conducts the process.
Concurrent Transformative Qualitative and Quantitative Integrated, embedded or connected Explicit The procedures for collection of qualitative and quantitative data are concomitant and guided by an explicit theoretical perspective.

Source: Creswell (2010).

Philosophically and methodologically, the method under investigation is based on pragmatism, which encompasses almost all studies that support quantitative and qualitative research. According to Greene (2007), the “heart” of pragmatism (therefore, the mixed perspective) is to invite “mental models” to occupy the same investigative space and to establish a respectful dialogue in which the perspectives inform one another mutually in addition to collectively fostering a better understanding of the investigated phenomenon. Pragmatism involves a multiplicity of perspectives, theoretical assumptions, methodological traditions, data gathering techniques and indicators analysis, personalized understandings, and value commitments that constitute the elements of mental models. Therefore, Sampieri and Mendoza (2018) argue that, oftentimes, these two approximations of knowledge may seem contradictory, but the seemingly contradictory may actually be a complement.

Finally, it is important to mention some challenges inherent to the development of a mixed methods research. Creswell and Clark (2013) strongly recommend for the development of this type of experiment that researchers have experience with qualitative and quantitative methods in both data collection and analysis techniques to validate data. A second point to be emphasized is the fact that mixed methods research demands more time, resources and, consequently, more extensive efforts. In addition, the recent nature of the methodology makes it necessary to display the value of this particular investigative model.

Methodology

This article is a documentary investigation that aims to “[…] understand a certain reality not in its immediate realization, but indirectly through the analysis of documents made by men about it” (MENDES; FARIAS; NÓBREGA-THERRIEN, 2011, p. 32). According to the documentary research method, researchers can employ different printed or digital documents, such as books, magazines, academic writings, journals, notes, photographs and paintings. According to McCulloch (2011, p. 249), “a document may be defined briefly as a record of an event or process”.

Regarding the present investigation, the documents of interest were limited to theses of a particular postgraduate program, published in the last six years (2013-2018). The search was conducted on the Sucupira Platform. As shown in Table 1, 104 theses from that Program were identified of which 11 (10.6%) employed mixed methods, thus being viable to the present analysis.

Table 1 – Search results 

TIME/YEAR THESES WRITTEN THESES WITH MIXED METHODS %
2013 20 03 15.0
2014 17 - -
2015 22 04 18.2
2016 14 02 14.3
2017 15 02 13.3
2018 16 - -
TOTAL 104 11 10.6

Source: https://sucupira.capes.gov.br/sucupira/

The phases after the search consisted of the literature review on mixed methods research, based on works of Creswell (2010), Creswell and Clark (2013), Sampieri, Collado and Lúcio (2013), Sampieri and Mendoza (2018), among others, and the analysis of the theses. The establishment of selection criteria for the theses was informed by the literature specific to the subject, especially the orientations from Creswell (2012) since he proposes the definition and a basic description of this approach. The author also lists questions that help researchers to design a mixed methods research (Chart 3).

Chart 3 – List of questions to design a mixed methods research 

1 Is the basic definition of mixed methods research provided?
2 Is the purpose of the study given, including both qualitative and quantitative purposes and the rationale for mixing methods?
3 Does the reader have a sense of the potential use of a mixed methods research strategy?
4 Are criteria identified for choosing a mixed strategy?
5 Is the mixed methods strategy identified and its criteria for selection given?
6 Is a visual model of the design given to clarify the research strategy?
7 Does the visual model include the correct notation?
8 Are data collection and data analysis procedures mentioned in relation to the strategy?
9 Are sampling strategies for both qualitative and quantitative data collection mentioned, and do they relate to the strategy?
10 Are procedures for validating both qualitative and quantitative data discussed?
11 Is the report structure detailed? Does it relate to the type of mixed methods strategy in use?

Source: Creswell (2012, p. 240).

In the chart, Creswell (2012) displays a list of questions to enable the development of a mixed methods design with the type of strategy, the specific procedures for collection, analysis and validation of data as well as the sample selection. He also includes a visual model of the design, the role of the researcher and the structure of the report. Hence, the selection of theses for the present investigation followed these assumptions. The next section identifies the methodological concerns of the authors of the selected theses regarding the fundamentals of the method they adopted.

Creswell’s questions displayed in Chart 1 worked, then, as a compass for the present analysis. The indicative of each of these questions are discussed in the next section for both qualitative and quantitatively, grouped in subsections by similarity and/or complementarity. In each subsection, the analysis of the theses were displayed to allow the identification of how they attend to the guidelines proposed by Creswell (2012) to design and structure a mixed methods research. Therefore, the reader can understand the main assumptions of a study of this nature in that particular Postgraduate Program as well as the aspects that were neglected/secondary.

To conclude these preliminary remarks, it is noteworthy that for ethical reasons the theses were named A, B, C,… since, according to Creswell (2012), researchers need to respect the research site, leaving them unaltered after an investigation given the possibility of prejudicial information being revealed during data collection.

The dialogue with/about the selected theses

Basic definition of mixed methods research and the reason to use concurrent qualitative and quantitative approaches

According to the information in Graph 1 below, five theses (45.5%) did not present the basic definition of mixed methods research (A, B, C, F and G), whereas the other six theses (54.5%) do present this definition according to the following authors: Minayo (2010) in thesis D; Minayo, Deslandes and Gomes (2007) in thesis E; Minayo and Sanches (1993) in thesis H; Minayo (1994) in thesis I; Minayo, Deslandes and Gomes (2007) in thesis J and Creswell (2012) in thesis K.

Graph 1 – Definition of mixed methods and explicit use of concurrent qualitative-quantitative strategySource: The authors, 2020. 

It is noteworthy that despite the emphasis given by Sampieri, Collado and Lúcio (2013, p. 550) to the existence of several classifications in the literature, such as “qual-quant research, integrative research, multimethod research, triangulation studies, multiple methods or mixed research”, among the investigated works, “qualitative-quantitative” was most recurrent, featuring in eight (72.7%) of the theses. One thesis (9.1%) was characterized as “mixed” and “primarily qualitative” was used in two works (18.2%) – even though they presented all characteristics of a mixed methods research, including descriptions of the quantitative dimension in the methodology chapter.

A different information in Graph 1 is that seven theses (63.6%) present justifications for the concurrent use of qualitative and quantitative approaches which underscore their complementary nature (see Chart 4). This perspective agrees with Sampieri and Mendonza (2008), who claim that mixed methods are the “third path”, as well as Creswell (2012), who highlights that qualitative and quantitative studies approach phenomena from different perspectives but, combined, can mitigate or even neutralize some disadvantages of these methods.

Chart 4 – Justification for the concomitant use of qualitative-quantitative approaches 

THESIS JUSTIFICATION REFERENCES
A Not presented
B
C
D The difference between approaches relies on their nature not on hierarchy, they are not incompatible. Minayo (2010)
E Complementary opposition Minayo (2007)
F Not presented
G The quantitative dimension enriches the investigation. Not presented
H The qualitative and quantitative aspects are not mutually exclusive. Minayo (1993)
I There is no quantification without qualification. Bauer; Gaskell (2002)
J The qualitative and quantitative approaches are complementary. Triviños (2010)
K Flick (2004)

Source: The authors, 2020.

Identification of procedures for data collection and analysis

It was verified that only thesis H failed to present procedures for data analysis. In the other theses (90.9%), the procedures for data collection and analysis were identified as seen in Chart 5.

Chart 5 – Procedures for data collection and analysis in the investigated theses 

THESIS DESCRIPTION DATA COLLECTION DATA ANALYSIS
A NOT PRESENT Multivariate statistical methods (ESCOFIER; PAGÉS, 1992) Correspondence analysis with Statistical Package for the Social Science (SPSS)
B Conclusive and ethnographic Structured interview; in-depth semi-structured interview; documentary analysis
  • Content analysis

  • Statistical analysis

C Descriptive and documentary Microdata of the analyzed institutions and national and international documents Statistical analysis with LibreOffice and index number base 100
D Exploratory and case study Documentary analysis; open questionnaire Content analysis; Likert scale
E Exploratory and case study Mixed questionnaire; conversation circle; semi-structured interview; documentary analysis; systematic observation Textual discourse analysis using Design Research
F Bibliographic and documentary Documentary analysis
  • Content analysis

  • Statistical analysis

G Exploratory and case study of the Multiple type In-depth interview; indexes of the institution (numeric data) and documentary analysis
  • Content analysis

  • Statistical analysis

H Exploratory, bibliographic and documentary Documentary analysis Neither qualitative or quantitative analysis procedures are mentioned but much of the data is presented in percentages
I NOT PRESENT Questionnaire (Google Drive); Memorial Content analysis; Likert scale
J Exploratory and cartographic Questionnaire (Google Drive); participant observation; open interview; field journal
  • Software NVivo

  • Queries

  • Graphs

K NOT PRESENT In-depth interview; documentary analysis Multiple regression statistical analysis; Analysis of variance (ANOVA); Tukey test; Propositional discourse analysis (content analysis)

Source: The authors, 2020.

The instruments to collect indicators express similarities, since most of them employ questionnaires, interviews and documentary analysis. It is noteworthy that researchers identified the focus adopted for each instrument, whether qualitative or quantitative – in accordance to Creswell (2010). According to this author, it is important that researchers identify and specify the types of data to be collected in the investigation, both qualitative and quantitative.

The present analysis also identified that theses A, I and K do not characterize the research clearly regarding both nature and method (see Chart 5). The other theses are mostly exploratory (five; 45.5%) with distinct methods, such as ethnographic, cartographic, case studies, bibliographic and documentary research.

Regarding data analysis, it was noted that “content analysis” – and its variations – was the most recurrent procedure in the theses, featuring in six theses (54.5%; B, D, F, G, I and K), always associated with quantitative instruments, which include software that helps researchers to treat greater volume of data such as SPSS (used in thesis A). This “statistical package” allows statistical and graph analysis with a wide range of indicators that work mainly in the preparation and validation of data; decision tree; regression models; advanced statistical patterns; tables; trends; categories; geospatial analysis and simulations (SANTOS, 2019).

Separate qualitative analyses (content analysis – Laurence Bardin) and quantitative analysis (statistical analysis) were conducted in thesis B, in addition to the use of correlation matrix, which, according to Creswell (2010), enable the combination of both quantitative and qualitative analyses. In this particular case, the treatment of data from this matrix resorted to statistical analysis, however the author does not explain which instrument was used in this specific treatment. This same situation is repeated in theses F and G. Another aspect that has already been mentioned is that procedures for data analysis (quantitative and qualitative) were not mentioned for thesis H.

An interesting feature in regard to the data analysis procedures was found in thesis E, which used Design Research. According to Manson (2006), this is a set of analytical techniques that employs a theoretical framework to design artifacts and uses specific techniques to evaluate these artifacts and to improve that theoretical framework at the end of the cycle. Therefore, based on an initial design, the researchers coordinate the activities with the participants, test, redesign and implement interventions to perfect the design. As the research progresses and a new “cycle” begins, the original design is perfected as well as the actions and interventions, which improves both practice and theoretical objectives. This is a creative trial and error strategy in which the solutions are reaffirmed and perfected until they are powerful enough to constitute a complete solution for the problem in case.

Identification of the sampling criteria and validation procedures for qualitative and quantitative data

Minayo (2017) stresses the importance of sampling in relation to the methodological credibility of an investigation. Indeed, the present study verified that all the theses in this study presented criteria for sample selection, as shown in Graph 2.

Graph 2 – Definition of criteria for sample selection and data validationSource: The authors, 2020. 

It is pivotal to state that only thesis B determined the classification of the sampling procedure. Two criteria were used for sample selection – one for the qualitative phase of the investigation and one for the quantitative phase. First, a sample representing the total of subjects in the group under investigation was selected (stratified probability sampling), then random sampling was employed to select the components of the qualitative data sample (simple random probability sampling). This thesis, then, uses probability sampling.

The other ten theses (90.9%; A, C, D, F, G, H, I and J) did not include this level of detail, however, upon reading them, it was clear that they did not use probability samples, but already established criteria based on the research objectives. It was also observed that theses E and K used non-probability convenience sampling (GIL, 2002), given that the subjects were invited to participate in the study based on their own interest.

Concerning the validation procedures, according to Graph 2, only four of the theses (36.4%; B, G, I and K) mention or clearly state these procedures for either qualitative or quantitative data. The authors of theses B, G and K refer to the validity of one collection instrument to highlight that the interviews were pre-tested for possible adjustments. Nonetheless, thesis I validated a questionnaire for which pre-test was also necessary to allow for the semantic analysis of questions and for adjustments.

It is noteworthy that validation of findings in the theses involved only one of the methods (either qualitative or quantitative) and failed to encompass all the phases of the study as recommended in the literature. Creswell (2012) and other scholars dedicated to mixed methods research defend the validation of findings for qualitative and quantitative phases of mixed research. Hence, for qualitative indicators, for example, researchers need to mention the strategies used to verify the accuracy of findings, which can include triangulation of sources, component testing, detailed descriptions among others.

Identification of mixed methods strategies and selection criteria

Creswell (2012) argues that adopting a strategy6 for mixed methods research demands a series of decisions to be made in relation to the approach that best suits the research problem; time distribution; combination and weight of the qualitative and quantitative methods; the visual model; the strategy and the approach which the researcher feels most comfortable using. Then, the criteria for choosing one strategy over the others is essential to demonstrate the methodology of a mixed methods research. Therefore, in this section, we focused on whether the authors identified the criteria and the strategies adopted in each of the theses.

It was verified that only thesis K (9.09%) identified the research strategy and the criteria adopted. This thesis relied on Creswell (2010, p. 250) to identify the research strategy and classified it as concurrent triangulation, defined as: “[...] the researcher collects concurrently both quantitative and qualitative data and then compares the two data bases to determine if there is convergence, differences or any combination [...]”. In the present case, the qualitative and quantitative findings are compared to the national educational data base.

The justification for that strategy relied on the research goals, which, as the authors point out, need two approaches (qualitative and quantitative) to be achieved as well as the data triangulation to favor a broader understanding of the phenomena. These perspectives agree with Creswell and Clark (2013), who emphasize that research goals are better achieved by employing mixed methods and that researchers need to be sensible enough to be aware of that.

Presenting a visual model for the research strategy, including the correct notation

It was observed that none of the theses in this research presented a notation or clear description of a visual model. To illustrate this fact, however, a visual model (Figure 1) was designed for the methodology in the only thesis that identified the research strategy: thesis K, as discussed in the previous section.

Figure 1 – Visual model with notation, adopted in thesis K – concurrent triangulation strategySource: The authors based on Creswell (2012) and the methodology used in thesis K. 

In this case, the concurrent triangulation was employed as a research strategy, based on the methodology described by the researcher and Creswell (2012). The visual model illustrates this thesis, since the qualitative and the quantitative data have equal weight (QUAL/QUAN) are collected and analyzed concurrently through the measurement of educational indicators (QUAN) and in-depth interviews (QUAL). Then, results are analyzed and compared to one another and to the national educational data base.

Final Remarks

The paradigmatic orientation of mixed methods research is recent. In addition, it is challenging for researchers to employ qualitative and quantitative methods in the same research. However, the mixed methods scientific production has increased and its popularization is a fact in all fields.

Nonetheless, the analyses of the theses written for a Postgraduate Program in Education, score seven according to the CAPES standard, revealed the need for doctoral students to become more familiarized with the most recent literature regarding the theory and methodology of mixed methods research. This is due to the novelty and the sudden popularization of mixed methods investigation, which leads to the modification/evolution of this research method rather quickly. Postgraduate students who wish to develop this type of study need to read the current literature and, above all, consider the contribution of international experiments, since scholars such as John W. Creswell, R. Burke Johnson, Anthony J. Onwuegbuzie, Roberto Hernández Sampieri, among others, are reference in the method.

At first, it was verified that mixed methods research is little employed in the Postgraduate Program in the present investigation, since only 11 (10.6%) of its 104 theses employed this method. Furthermore, only one thesis used the name “mixed”, whereas eight theses (72.7%) were labeled “qualitative-quantitative” and two were classified as “predominantly qualitative”, despite showing characteristics of a mixed methods investigation, including descriptions of a quantitative nature in the methodology chapter. Also, five theses (45.5%) did not present a clear definition of mixed methods research and seven (63.6%) justified the concurrent use of qualitative and quantitative methods, underscoring the complementarity (not the opposition) of these methods.

Five of the eight theses stated clearly the exploratory nature of the research and the use of several research methods – such as ethnography, cartography, bibliographic method, documentary research and case study. Except for thesis H, the other ten theses (90.9%) indicated the instruments of data collection and analysis, distinguishing which were related to the qualitative and quantitative phases of the research. The most recurrent instruments were the questionnaire, the interview and documentary research. Six theses (54.5%) relied on “content analysis” – and variations – followed by quantitative procedures, such as specific software, Likert scale, among others. Nonetheless, three theses (B, F and G) mentioned statistical analysis without specifying the instrument or how these analyses were conducted.

All the theses presented the criteria for sample selection but only thesis B detailed the sampling technique, which was classified as probability sampling. The sampling technique used in the other theses were of the non-probability type. Regarding validation of findings, only four theses (36.4%) made clear mention to the topic, but even those were not in total accordance with Creswell (2012). This author points out that validation procedures for all types of data (qualitative and quantitative) must be described, however, these theses only described one of the methods (either qualitative or quantitative).

Just one of the theses clearly defined the mixed methods research strategy adopted and the selection criteria. On the other hand, none of the theses displayed a visual model for their strategy nor used the correct notation. These results reinforce the need for researchers in that Program to search for more up to date literature regarding mixed methods research, since many scholars, such as de Tashakkori and Teddlie (1998), Sampieri, Collado and Lúcio (2013), Johnson and Onwuebuzie (2004), share the understandings and guidelines proposed by Creswell (2012) – the author of reference for the present analysis – regarding the features of a mixed methods investigation.

Another point to be stressed is that there is a noticeable theoretical and methodological evolution in methodological path developed in the theses that used mixed methods in the last six years of the Program (from 2013 to 2018). This is evident since the most recent theses make reference to the international literature and display greater intertwining of qualitative and quantitative features, in addition to more grounded justifications.

To conclude, it is worth mentioning the possibility of readers perceiving, to greater or lesser degrees, the potential use of mixed methods designs (CRESWELL, 2012), which is evident in the temporal evolution of this perception, as stated above. The most recent theses present clearer perspectives and justifications to how qualitative and quantitative methods were used in the course of the investigation.

It is also necessary to underscore the relevance of the present study for Brazilian postgraduate programs in education and for mixed methods research, with particular emphasis to the methodological rigor that is a key element of support for the scientificity of an investigation. In addition, the aim of trying to locate the gaps in the use of mixed methods designs is to orient/help researchers who wish to use this approach, and, at the same time, contribute to the dissemination of mixed methods researches.

REFERENCES

BAUER, Martin W.; GASKELL, George. Pesquisa qualitativa com texto, imagem e som: um manual prático. Petrópolis: Vozes, 2002. [ Links ]

CRESWELL, John W. Educational research: planning, conducting, and evaluating quantitative and qualitative research. 4. ed. Boston: Pearson, 2012. [ Links ]

CRESWELL, John W. Projeto de pesquisa: método qualitativo, quantitativo e misto. 3. ed. Porto Alegre: Artmed, 2010. [ Links ]

CRESWELL, John W. Qualitative inquiry & research design: choosing among five approaches. 3. ed. Thousand Oaks: Sage, 2013. [ Links ]

CRESWELL, John W.; CLARK, Vick L. Plano. Pesquisa de métodos mistos. 2. ed. Porto Alegre: Penso, 2013. [ Links ]

CRESWELL, John W. et al. Advanced mixed methods research designs. In: TASHAKKORI, Abbas M.; TEDDLIE, Charles B. (ed.). Handbook of mixed methods in social and behavioral research. Thousand Oaks: Sage, 2003. p. 209-240. [ Links ]

FLICK, Uwe. Uma introdução à pesquisa qualitativa. 2. ed. Porto Alegre: Bookman, 2004. [ Links ]

GALVÃO, Maria Cristiane Barbosa; PLUYE, Pierre; RICARTE, Ivan Luiz Marques. Métodos de pesquisa mistos e revisões de literatura mistas: conceitos, construção e critérios de avaliação. InCID, Ribeirão Preto, v. 8, n. 2, p. 4-24, set. 2017/fev. 2018. Disponível em: https://www.revistas.usp.br/incid/article/view/121879/133611. Acesso em: 10 jun. 2019. [ Links ]

GIL, Antonio Carlos. Como elaborar projetos de Pesquisa. 4 ed. São Paulo: Atlas, 2002. [ Links ]

GREENE, Jennifer C. Mixed methods in social inquiry. San Francisco: Jossey-Bass, 2007. [ Links ]

IVANKOVA, Natalyia V. Mixed methods applications in action research: from methods to community action. Thousand Oaks: Sage, 2015. [ Links ]

JOHNSON, R. Burke; ONWUEGBUZIE, Anthony J. Mixed methods research: a research paradigm whose time has come. Educational Researcher, Washington, DC, v. 33, n. 7, p. 14-26, 2004. Disponível em: https://journals.sagepub.com/doi/10.3102/0013189X033007014. Acesso em: 23 fev. 2019. [ Links ]

MANSON, Neil J. Is operations research really research? Orion, v. 22, n. 2, p. 155-180, 2006. Disponível em: https://orion.journals.ac.za/pub/article/view/40. Acesso em: 20 dez. 2019. [ Links ]

McCULLOCH, Gary. Historical and documentary research in education. In: COHEN, Louis; MANION, Lawrence; MORRISON, Keith. Research methods in education. 7. ed. London: Routledge, 2011. p. 248-255. [ Links ]

MENDES, Emanoela Therezinha Bessa; FARIAS, Isabel Maria Sabino de; NÓBREGA-THERRIEN, Silvia Maria. Trabalhando com materiais diversos e exercitando o domínio da leitura: a pesquisa bibliográfica e a pesquisa documental. In: NÓBREGA-THERRIEN, Silvia Maria; FARIAS, Isabel Maria Sabino de; NUNES, João Batista Carvalho. Pesquisa científica para iniciantes: caminhando no labirinto. v. 3. Fortaleza: UECE, 2011. p. 25-42. [ Links ]

MINAYO, Maria Cecília de Souza. Amostragem e saturação em pesquisa qualitativa: consensos e controvérsias. Revista Pesquisa Qualitativa, São Paulo, v. 5, n. 7, p. 1-12, abr. 2017. Disponível em: https://edisciplinas.usp.br/pluginfile.php/4111455/mod_resource/content/1/Minayosaturacao.pdf. Acesso em: 20 jun. 2020. [ Links ]

MINAYO, Maria Cecilia de Souza. Ciência, técnica e arte: o desafio da pesquisa social. In: MINAYO, Maria Cecilia de Souza (org.). Pesquisa social: teoria, método e criatividade. Petrópolis: Vozes, 1994. p. 9-29. [ Links ]

MINAYO, Maria Cecília de Souza. O desafio do conhecimento. São Paulo: Hucitec, 2010. [ Links ]

MINAYO, Maria Cecília de Souza; DESLANDES, Suely Ferreira; GOMES, Romeu. Pesquisa social: teoria, método e criatividade. 26. ed. Petrópolis: Vozes, 2007. [ Links ]

MINAYO, Maria Cecília de Souza; DESLANDES, Suely Ferreira; GOMES, Romeu. Pesquisa social: teoria, método e criatividade. 28. ed. Petrópolis: Vozes, 2009. [ Links ]

MINAYO, Maria Cecilia de Souza; SANCHES, Odécio. Quantitativo-qualitativo: oposição ou complementaridade? Cadernos de Saúde Pública, Rio de Janeiro, v. 9, n. 3, p. 237-248, set. 1993. Disponível em: http://www.scielo.br/scielo.php?script=sci_a rttext&pid=S0102-311X1993000300002&lng=en&nrm=iso. Acesso em: 20 abr. 2021. [ Links ]

SAMPIERI, Roberto Hernández; COLLADO, Carlos Fernández; LUCIO, Maria del Pilar Baptista. Metodologia de pesquisa. 5. ed. São Paulo: McGraw-Hill, 2013. [ Links ]

SAMPIERI, Hernández Roberto; MENDOZA, Paulina Christian. Metodología de la investigación: las rutas cuantitativas, cualitativas y mixta. 1. ed. México, DC: McGraw-Hill, 2018. [ Links ]

SANTOS, Alexandre. IBM SPSS como ferramenta de pesquisa quantitativa. São Paulo: PUC, 2019. Disponível em: https://www.pucsp.br/sites/default/files/download/posgraduacao/programas/administracao/IBM-SPSS-como-ferramenta%20de-pesquisa-quantitativa-alexandra-santos.pdf. Acesso em: 10 jul. 2019. [ Links ]

TASHAKKORI, Abbas M.; CRESWELL, John W. The new era of mixed methods. Journal of Mixed Methods Research, Michigan, v. 1, n. 3, p. 4-7, 2007. Disponível em: https://journals.sagepub.com/doi/pdf/10.1177/2345678906293042. Acesso em: 12 jun. 2020. [ Links ]

TASHAKKORI, Abbas M.; TEDDLIE, Charles B. Mixed methodology: combining qualitative and quantitative approaches. London: Sage, 1998. (Social research methods series; 46). [ Links ]

TRIVIÑOS, Augusto Nibaldo Silva. Introdução à pesquisa e ciências sociais: a pesquisa qualitativa em educação. 1. ed. São Paulo: Atlas, 2010. [ Links ]

YIN, Robert K. Estudo de caso: planejamento e métodos. 3. ed. Porto Alegre: Bookman, 2005. [ Links ]

* The translation of this article is responsibility of Larissa de P. Cavalcanti. The translator and the author take full responsibility for the translation of the text, including titles of books/articles and the quotations originally published in Portuguese.

1- Disponibilidade de dados: todo o conjunto de dados que dá suporte aos resultados deste estudo foi publicado no próprio artigo.

6- Chart 2 summarizes the main strategies for mixed methods research according to Creswell (2010).

Received: September 20, 2020; Revised: November 24, 2020; Accepted: February 10, 2021

Luciana Rodrigues Leite is a PhD student in the Postgraduate Program in Education at Universidade Estadual do Ceará (PPGE/UECE). She is an associate professor level D for the Course of Chemistry at Universidade Estadual Vale do Acaraú (UVA), Sobral, Ceará, Brazil.

Ana Paula dos Santos Reinaldo Verde is a PhD student in the Postgraduate Program in Education at Universidade Estadual do Ceará (PPGE/UECE). She is a school teacher in the State of Maranhão, São Luis-MA, Brazil.

Francisco das Chagas Rodrigues de Oliveira is a PhD student in the Postgraduate Program in Education at Universidade Estadual do Ceará (PPGE/UECE). He is a school teacher for the Municipal School System of Boa Viagem and Monsenhor Tabosa, Ceará, Brazil.

João Batista Carvalho Nunes holds a PhD in Philosophy and Science of Education by the Universidad de Santiago de Compostela (Spain). He is a professor for the Postgraduate Program in Education at Universidade Estadual do Ceará (PPGE/UECE), Fortaleza, Ceará, Brazil.

Creative Commons License  This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.