INTRODUCTION

Data envelopment analysis (DEA) is a multivariable technique based on non- -parametric mathematical models which seeks to evaluate the performance of decision-making units (DMU). This is done by comparing similar units which differ in the quantities of resources employed (inputs) and results achieved (outputs) (^{CORRÊA et al., 2019}; FERREIRA; GOMES, 2020).

According to ^{Ferreira and Gomes (2020)}, DEA developed mainly from Michael James Farrell’s article, “The Measurement of Productive Efficiency”, published in 1957 in the Journal of the Royal Statistical Society, in which the author “[…] tried to develop better methods for evaluating productivity through concepts of activity analysis” (FERREIRA; GOMES, 2020, p. 17).

However, ^{Charnes, Cooper and Rhodes (1978)} say that, in fact, the history of DEA began with Edwardo Lao Rhodes’ PhD thesis, supervised by William W. Cooper (^{RHODES, 1978}). Briefly, Rhodes’ research sought to develop a methodology that could compare the efficiency of American public schools, considering their participation in Follow Through, a federal program for the educational integration of vulnerable children (^{UNITED STATES, 1975}; ^{RHODES, 1978}).

Since then, DEA-related concepts were extended and enhanced, consolidating it as a widely adopted methodological approach in school effectiveness studies (AGASISTI; BONOMI; SIBIANO, 2014), which, according to ^{Johnes (2006)}, reflects the simplicity of its application to contexts with multiple inputs and outputs, as is typically the case with education. The theoretical foundation for these studies is that the school’s inputs are key for achieving its educational outputs (BROOK; SOARES, 2008; ^{LIMA, 2011}).

DEA has two most popular approaches: the one which requires a constant return to scale (CRS), and the one which demands a variable return to scale (VRS). In practice, the main difference between these is related with the proportionality axiom. Contrary to the VRS model, the CRS model considers that any variation in the inputs implies a proportional variation in the outputs (^{SANTOS; FREITAS; FLACH, 2015}; ^{SOUZA; GOMES, 2016}).¹

For these approaches, in order to determine the efficient frontier, two radial orientations are possible: orientations to inputs (i.e., input-oriented approach) and orientations to output (i.e., output-oriented approach). The former deals with the amount of inputs that can be reduced, provided that outputs remain the same. The later deals with maximizing outputs, though without changing the amount of inputs.

However, these classic models have originated variations, like the ones that consider fuzzy coefficients (^{GUO; TANAKA, 2001}; ^{LERTWORASIRIKUL et al., 2003}), windows analysis (^{CHARNES et al., 1984}), cross-efficiency models (SEXTON; SILKMAN; HOGAN, 1986) and methods with stochastic variables, or chance- -constrained programming (CCP) (^{SENGUPTA, 1987}).

These variations seek to offset some disadvantages to the DEA model, like those arising from its non-parametric nature which prevent more robust statistical analyses. However, procedures with two or more stages have also been conducted using statistical inference techniques to complement results obtained via DEA.

In this strand, and considering DEA’s dimension and importance in education research, the goal of this article is to investigate, by means of a systematic literature review, how this technique has been used in studies on school effectiveness. The idea is therefore to answer the following questions: which DEA approach and which input and output variables are most frequently used in the investigated studies? How have the authors of these studies overcome DEA’s limitations, particularly those resulting from the non-supply of conventional measures of statistical significance and from the non-indication of strategies that might help to improve education quality?

To this end, besides this introduction, the text is divided into another four sections. In the first, the method used to select the articles that form this review is detailed. In the second, results are presented, providing a summarized description of the studies included in it. The third presents a discussion originating from the analysis of the mapped studies. Finally, in the fourth section, the final considerations are outlined.

METHOD

In order to select the scientific articles that make up this review, the preferred reporting items for systematic reviews and meta-analyses (Prisma)² model was followed. Prisma was developed by way of a checklist in order to provide authors with a guide on systematic reviews and meta-analyses that helps with the process of eligibility and analysis of sources (^{MOHER et al., 2010}; ^{PAGE et al., 2021}).

The article selection process was carried out in March 2021 by searching in the Coordination for the Improvement of Higher Education Personnel’s (Capes) Journal Portal, with expanded access to the Scopus and Web of Science (WoS) electronic databases, which were chosen for being the main indexing databases considered by the Qualis Journals rating system in education, according to a Capes technical report³ (^{COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR - CAPES, 2019}). Moreover, this study assumes that the importance of these bases for enhancing the dissemination and circulation of scientific publications is well established.

To this end, the descriptors ‘Data Envelopment Analysis’ AND ‘School Effectiveness’ were used, and the selection was limited to articles published between 2010 and 2019 in order to obtain recent studies related with the subject, and because this period was considered sufficient for all articles in this period to be indexed in the aforementioned databases. To be selected, articles had to be peer-reviewed, and their full texts had to be available on the internet at the Capes Journal Portal through the Federate Academic Community (CAFe).

The pre-selected articles were read by the authors, and studies whose focus was not directly related with school effectiveness or which did not address basic education specifically were excluded. For the other studies, the input and output variables were analyzed; the works were grouped through a procedure based on thematic analysis (^{BRAUN; CLARKE, 2006}), considering also the employment of the DEA approach, and thus the selected studies were described.

PRESENTATION OF RESULTS

Initially, 28 scientific articles were identified, eight of which were indexed in at least two of the databases considered. By excluding those that repeated in different bases, there were 20 studies left for initial consultation. Of these, five were excluded, as their full texts were not available at the Capes Journal Portal through the CAFe, with the other 15 being considered as potentially available for inclusion in the review. However, after reading their full texts, another five works were excluded, as they did not address the proposed subject, leaving ten articles for the bibliographic research.

Figure 1 represents the flowchart of the selection of articles for analysis, based on Prisma guidelines. It should be noted the authors agreed 100% throughout this eligibility process, which adds to the credibility of the selection.

Source: The authors.

FIGURE 1 Flowchart of the selection of articles based on Prisma guidelines 

Chart 1 presents bibliographic information on these articles, i.e., their years, the journals in which they were published, their titles, authors, and DEA approach.

As seen in Chart 1, while the selection does not comprise a massive output, at least one article was published per year (except for 2011 and 2017) for the 2010-2018 period. These works were published in seven different journals, with Quality Assurance in Education (QAE) and the International Journal of Educational Management (IJEM) being the only journals featuring more than representation - two and three articles, respectively.

As for the DEA approach used, three studies chose CRS, with two of these adopting orientations to output (SARRICO; ROSA; COELHO, 2010; YALÇIN; TAVŞANCIL, 2014) and one giving no information on this respect (MUNOZ; QUEUPIL, 2016). As to those using the VRS approach, five studies were found, four of which employ orientations to output (PORTELA; CAMANHO; KESHVARI, 2013; ^{AGASISTI, 2013}; AGASISTI; BONOMI; SIBIANO, 2014; AGASISTI; ZOIDO, 2018), and one to input (HUGENIN, 2015). In addition, one study uses the cross-efficiency model (GOURISHANKAR; LOKACHARI, 2012) and one does not specify the approach used (HALKIOTIS; KONTELES; BRINIA, 2018).

With a view to identifying the variables used as inputs in these studies for explaining school performance, Table 1 below was built.

As for outputs, five articles consider outcomes in national standardized exams with a focus on vernacular language and mathematics (PORTELA; CAMANHO; KESHVARI, 2013; AGASISTI; BONOMI; SIBIANO, 2014; ^{HUGUENIN, 2015}; MUNOZ; QUEUPIL, 2016; HALKIOTIS; KONTELES; BRINIA, 2018).

Another three articles appropriate outcomes in the Programme for International Student Assessment (PISA) (^{AGASISTI, 2013}; YALÇIN; TAVŞANCIL, 2014; AGASISTI; ZOIDO, 2018). In common, these articles focus on outcomes in mathematics. Moreover, ^{Agasisti (2013)} and ^{Yalçin and Tavşancil (2014)} also consider the sciences assessment, and ^{Yalçin e Tavşancil (2014)} and ^{Agasisti and Zoido (2018)}, the vernacular language assessment.

The other two articles analyzed consider as output information from internal exams. In addition, ^{Sarrico, Rosa and Coelho (2010)} include student flow indicators (dropout and completion), and ^{Gourishankar e Lokachari (2012)}, gross enrollment rate and the percentage of students admitted to universities or technology institutes.

For a better description of these articles, the three subsections below describe briefly the analyses conducted in the mapped studies. To this end, and based on thematic analysis (^{BRAUN; CLARKE, 2006}), groups were formed according to the DEA approach used in each work. The first two subsections describe the three and five articles that assume, respectively, constant and variable returns to scale, while the third addresses the other works.

DISCUSSION

In the mapped studies, no consensus was found between authors regarding the choice of variables to be used as inputs in the application of DEA models, which can be explained by the diversity of realities, cultures and populations investigated. In this respect, it is noteworthy that two articles analyze schools located in Portugal (SARRICO; ROSA; COELHO, 2010; PORTELA; CAMANHO; KESHVARI, 2013) and two in Italy (^{AGASISTI, 2013}; AGASISTI; BONOMI; SIBIANO, 2014). Another five approached data for India (GOURISHANKAR; LOKACHARI, 2012), Turkey (YALÇIN; TAVŞANCIL, 2014), Switzerland (^{HUGUENIN, 2015}), Chile (MUNOZ; QUEUPIL, 2016) and Greece (HALKIOTIS; KONTELES; BRINIA, 2018). The work of ^{Agasisti and Zoido (2018)} was the only one that considered a comparative study between countries.

As indicated in Table 1, most of these variables were used in a single study. On the other hand, there is a predominance of classic factors, widely discussed in the specialist literature because of their significant impacts on education quality, which assume conditions intrinsic to the classroom, to the school context and to students’ lives.

As for the variables used as outputs in the application of DEA models, the massive majority of studies (n = 8) considered outcomes in standardized exams (national or international), with priority to some contents - vernacular language and mathematics -, which reinforces the hypothesis of hierarchization of these curriculum components at the international level for purposes of measuring or monitoring basic education quality.

It should be noted that the use of DEA in studies on school effectiveness has technical and practical advantages in relation to parametric analyses, particularly because it does not impose a particular functional form on the link between inputs and outputs for comparing schools. In addition, no relation between them is required, and its application does not demand a specific distribution of data in the population.

However, due to its non-parametric nature, DEA is not able to explain the effects of the analyzed factors, for purposes of school effectiveness, which is a major limitation of this technique. Because of this characteristic, no conventional measures of statistical significance can be provided, which prevents inferences for the target-population that might widen the scope of results.

To bypass this problem, most of the mapped articles (n = 6) perform a regression analysis as a second-stage procedure, complementing the results from DEA. Of these, three use Tobit regression (^{AGASISTI, 2013}; AGASISTI; BONOMI; SIBIANO, 2014; AGASISTI; ZOIDO, 2018) and another three, OLS regression (GOURISHANKAR; LOKACHARI, 2012; ^{HUGUENIN, 2015}; HALKIOTIS; KONTELES; BRINIA, 2018) in order to investigate the factors affecting schools’ efficiency. This made it possible to examine how the different outputs of school units can be attributed to environmental and managerial factors.

Thus, it was found that when students’ socioeconomic status was used as an independent variable in the regression model, results indicated that schools that served a greater proportion of vulnerable students were more inefficient. Indeed, this factor has become almost unanimous in terms of impact on school outcomes in standardized assessments, considering studies conducted for that purpose at the international level, whose initial milestone was the release of the Coleman Report⁴ (^{COLEMAN, 1966}). In addition, factors related with units’ management also showed statistical significance in the analyses conducted in the mapped studies.

On the other hand, the analyses that considered units’ infrastructure, geographic factors, class size and the proportion of female students did not present results in a single direction, which indicates that the effects of these variables on school efficiency can take different courses for studies developed in different places. Thus, no uniformity was found in conclusions about the influence of these factors on units’ estimated performance.

It should also be noted that bootstrapping was applied to DEA in three studies developed from a sample of the analyzed target-population (^{AGASISTI, 2013}; AGASISTI; BONOMI; SIBIANO, 2014; AGASISTI; ZOIDO, 2018), which allowed to correct the estimated efficiency values, considering the random errors in the sampling process. This allowed estimating, for each analyzed unit, the confidence interval and bias, in addition to the corrected efficiency, thus making it possible for statistical inferences to be made from results.

As for the DEA approaches, VRS was most used, as it was employed in five studies (^{HUGUENIN, 2015}; PORTELA; CAMANHO; KESHVARI, 2013; ^{AGASISTI, 2013}; AGASISTI; BONOMI; SIBIANO, 2014; AGASISTI; ZOIDO, 2018). These results are in line with Faria, Jannuzzi and Silva (2008, p. 164), who found that “[…] in general, the relations established in the field of public policy do not suppose constant returns to scale”.

With regard to radial orientation, six studies chose output-oriented models (SARRICO; ROSA; COELHO, 2010; PORTELA; CAMANHO; KESHVARI, 2013; ^{AGASISTI, 2013}; AGASISTI; BONOMI; SIBIANO, 2014; YALÇIN; TAVŞANCIL, 2014; AGASISTI; ZOIDO, 2018), one, an input-oriented model (^{HUGUENIN, 2015}), and one considered both orientations (HALKIOTIS; KONTELES; BRINIA, 2018). Indeed, still according to ^{Faria, Jannuzzi and Silva (2008)}, those relations established in the field of public policy require maximizing outputs from the same inputs (output- -oriented), assuming limited budget funds for the area, which must be used in the most efficient way possible.

The studies that propose a longitudinal analysis also use complements to DEA methodology in order to achieve their goals, whether by adapting the Malmquist index (PORTELA; CAMANHO; KESHVARI, 2013) or through windows analysis (YALÇIN; TAVŞANCIL, 2014). While they are few in this mapping, studies of this type are very important for monitoring education quality and preventing unfavorable alterations in schools’ efficiency.

It is also necessary to highlight a limitation found in all mapped articles which regards their methodological structure. The authors are concerned with estimating units’ efficiency through their outcomes in standardized tests, school performance and quality indicators, disregarding the existence of the multiple processes occurring within schools which are not measurable.

These processes constitute practices and subjectivities that can only be understood by going into these units’ daily life, which is not done by any of the mapped studies, although two of them (SARRICO; ROSA; COELHO, 2010; GOURISHANKAR; LOKACHARI, 2012) consulted the analyzed units. According to ^{Alves and Garcia (2002}, p. 261), in order to understand these processes, it is necessary to feel the school, and not just look at it “[…] in a superior attitude, from a distance”, which might have been done through in loco studies, interacting with the school community, thus leading to more significant, adequate results.

FINAL CONSIDERATIONS

This article aimed to investigate how DEA has been used in studies on school effectiveness. To this end, a systematic literature review was carried out following Prisma guidelines (^{MOHER et al., 2010}; PAGE, 2021) and using thematic analysis (^{BRAUN; CLARKE, 2006}). Studies of this type, which analyze a body of research, not only enable the dissemination of research already conducted, but can also be used as tools to assist researchers with regard to research trends on a particular subject.

In the case of this article, the review focused on education research, thinking the school as the object of analyses in the mapped articles. While the reviewed output is not massive, considering the evaluated period (2010-2019), it allowed identifying potentials and limitations to the application of methodology involving DEA in these studies.

Regarding the potentials, it is noteworthy that the application of DEA does not require the existence of functional relations between the output and input variables, nor is it necessary to pre-assign weights to them. Besides, the ease of application and interpretation of the techniques involving it, particularly given the extensive development of specialist computer software, makes its use attractive in contexts that require a practical study, though without disregarding scientific rigor.

As for the limitations found, it is noteworthy that they stem from DEA’s non- -parametric nature, due to which no statistical inferences can be made for the study’s target population, nor can the scope of results be widened. However, bootstrapping was found to be useful in some of the mapped studies, and can be adopted as a way of bypassing this limitation.

It should also be noted, as a major limitation of its use, that DEA only identifies peer/benchmark units through comparison with units with similar characteristics to theirs, without even indicating the determinant factors for improving education quality. However, second-stage procedures, which encompass correlation and regression analyses, have proved of great relevance to bypassing this problem. On the other hand, no consensus was found about which type of regression to use, since part of the mapped studies use Tobit regression, while others use OLS regression.

In future studies, it seems interesting to investigate how Brazilian research about school effectiveness is being conducted, with a view to identifying the place of DEA as a methodological approach, since no studies carried out in the national context were mapped in this review. Finally, it is expected that this article serves as an instrument to assist and guide researchers of education who intend to incorporate DEA in their empirical analyses.