Introduction

In the city of Rio de Janeiro, schools in the early years of basic education went from an IDEB score of 4.6 in 2007 to 5.7 in 2017¹. This improvement, however, did not occur evenly across all schools. Some schools improved greatly, while others lagged behind, even though they are all part of the same school network and are all subject to the same legal rules. Although socioeconomic factors are relevant and can explain part of this asymmetry, there are situations in which other factors contribute to explain this inequality (^{Ragazzo; Almeida, 2020}). Given that we want everyone to have access to quality education, how can we replicate success stories across the network?

A recent research project, called Education with Equity, followed 215 schools with student bodies of low socioeconomic status in Brazil to identify which strategies worked best to reduce the gap between high and low-performing schools (^{Lemann Foundation; Itaú BBA, 2018}; ^{Faria; Guimarães, 2015}). The research’s results - in line with the broader academic consensus that we will explore later - point to the positive influence of good principals and good management practices at the local level. On the other hand, the research also highlights the problem caused by the existence of asymmetries within a single school system. In order to study this same type of problem, ^{Abrucio (2010)} paired schools in the state of São Paulo with similar student bodies but significant differences in academic performance. The research showed that leadership, including principals’ turnover rate and experience, were central factors in explaining academic discrepancies.

There is a high degree of agreement in the international academic community around the idea that principals exert an influence over educational outcomes. A recent review of the literature summarizes it, saying that “[...] there is substantial research evidence demonstrating that school leaders are a powerful driver of student outcomes” (^{Hernan et al. 2017}, p. 3). Previous work has investigated the causes of this influence, showing that several characteristics of principals correlate with student performance. Characteristics such as leadership style (^{Polon, 2009}), engagement with teachers (^{Oliveira; Waldhelm, 2016}), experience (^{Clark; Martorell; Rockoff, 2009}), and training (^{Gates et al., 2019}), among others, predict the success of students on standardized tests. Thus, it is important to use this knowledge to advance our public policy goals.

This strategy seems especially promising given the robustness of the association between principals and school results. While it is true that some of the literature investigates the performance of school principals in countries where they have broad discretion to hire and fire teachers, such as in the United States and the United Kingdom (^{Branch; Rishkin; Hanushek, 2013}; ^{Thody et al, 2007}), there are also several studies conducted in Brazil and Europe in which principals have limited discretion in administrative and pedagogical matters that demonstrate the same effect (^{Polon, 2009}; ^{Abrucio, 2010}, ^{Lemann Foundation; Itaú BBA, 2012}, ^{Faria; Guimarães, 2015}; ^{Oliveira; Waldhelm, 2016}; ^{Oliveira; Carvalho, 2018}; ^{Karstanje; Webber, 2008}, ^{Thody et al., 2007}).

The purpose of this paper is normative: we describe a proposal that we believe is capable of improving the problem of equity within large municipal school systems. More specifically, we believe that a principal training program that uses a peer-mentoring strategy is capable of improving the performance of the schools that consistently perform the worst. There are good reasons to explore the idea: governments around the world have invested in principal training programs, generally reaping rewards (^{Gates et al., 2019}; ^{Hernan et al., 2017}; ^{Clark; Martorell; Rockoff, 2009}; ^{Karstanje; Webber, 2008}; ^{Thody et al., 2007}; ^{Walker; Hallinger, 2015}). In a mismatch with the rest of the world and despite the academic consensus, however, there are few public policies focused on school principals in Latin America (a fact reported by ^{Vailant, 2011}). Brazil and the city of Rio de Janeiro are no exceptions in the region. At both the national and municipal levels, there is an absence of public policies aimed at school principals.

The asymmetric results of the schools in the Rio de Janeiro municipal network, coupled with a review of the literature on the influence of principals on learning performance in schools, means that this solution has high potential to produce a low-cost and effective public policy. However, there is reasonable difficulty in establishing a training format and, even more so, in identifying what can be subject to training, given that the problems are not homogeneous in the school network. To show the plausibility of our proposal, we will pursue the following strategy: (i) we will present a review of the international literature on training programs for principals, showing how mentoring programs have helped education networks abroad; (ii) we will discuss recent work that identifies pairs of schools geographically close but with discrepant performance; and, finally (iii) we will describe the results of novel empirical analyses showing that there are systematic differences between principals of schools with good performance and principals of schools at risk, to justify the regulatory measure.

Principal Training Programs: Mentoring as a Regulatory Approach

The knowledge that school principals matter should be exploited by implementing public education policies that aim to improve their performance. But how exactly should we do this? Among the many possibilities, what should the focus be? Governments and researchers around the world suggest that the answer lies in training programs for principals.

According to ^{Clark, Martorell, and Rockoff (2009, p. 4)}, in New York City alone, several of these programs have been implemented. The authors’ research showed mixed results about their influence, but later work points to the effectiveness of at least one of them (the Aspiring Principals Program) on several metrics. ^{Gates et al. (2019)} report that students in schools led by principals who came from the program – which consists of a three-week course followed by a one-year residency – perform significantly better, even when using strict comparison techniques with other schools, such as propensity score matching (^{Rosenbaum; Rubin, 1983}).

Principal training programs that combine academic knowledge with experiential learning are also common in Canada (^{Pedwell et al., 2010}), the United Kingdom (^{Gunter; Forrester, 2009}), continental Europe (^{Karstanje; Webber, 2008}; ^{Thody et al., 2007}), Asia (^{Walker; Hallinger, 2015}), and Australia (^{Dempster; Robson; Gaffney, 2010}), although there is less quantitative data on the outcomes of these experiences. In any case, the general tone of the articles reviewed is clearly positive. Despite this global receptivity to training principals as a public policy focused on improving school network performance, the practice is still neglected in Latin America (^{Vailant, 2011}, p. 583-584).

Given that principal training programs seem to be a good approach, what are the characteristics of an effective principal training program? One hallmark of the development programs surveyed is the inclusion of a mentoring system in addition to the more traditional methods of academic instruction (^{Gates et al., 2019}; ^{Walker; Hallinger, 2015}). Some evidence suggests that, in the United States, mentoring models are even superior to more traditional university training (^{Grissom; Harringon, 2010}). Thus, we propose that this hands-on learning dimension should be valued in principal training programs. This is especially true in places like Rio, where small geographic variations can cause big differences in the challenges schools face.

A good model for developing a school leadership program that respects this diversity comes from Ontario, Canada. Since 2003, one of the priorities of the Ontario government has been to improve the quality of education in the province, with a focus on school leadership (^{Pedwell et al., 2010}). One of the first efforts made by the province was to systematize the existing knowledge within the school system, which included the Principals Congress, initiated in 2007. The Principal’s Congress is an annual event focused on identifying and disseminating best practices within the school system (^{Leithwood et al., 2010}). Eventually, this knowledge was combined with academic expertise to create the Ontario Leadership Framework (^{Leithwood, 2012}), an evidence-based document that outlines the key dimensions of effective school leadership. Ontario’s strategies include a focus on underperforming schools and mentoring, with initiatives such as the Leader-to-Leader initiative, a project that “[...] brought together 11 principals from struggling schools with 11 principals from schools that had overcome difficult circumstances to share effective practices”, with great results (^{Pedwell et al., 2010}, p. 610).

Data-driven bottom-up strategy for public policies aimed at training principals

The literature review on principal training programs shows the effectiveness of this strategy, especially when the training involves mentoring programs. But how can we define which principals should be involved in such programs? Even more precisely: which principals can potentially be integrated into the program as mentors, and which should be encouraged to participate as mentees? The results reported in ^{Ragazzo and Almeida (2020)} help answer these questions.

The first step in the procedure laid out in ^{Ragazzo and Almeida (2020)} involves identifying benchmark schools and underperforming schools according to their performance over a given period of time. Some schools start out excellent and remain excellent throughout the period. These schools are good examples and at first glance seem to be the most obvious candidates to serve as benchmarks. However, if the goal is to ensure not only quality but also equity, they are less valuable than those that have undergone recent improvements. If the goal is to help the worst performing schools, it makes more sense to study cases of schools that used to be among the worst, but overcame difficulties to increase academic quality. In the case of Rio, schools were classified as either benchmark or at risk using a mix of absolute (initial IDEB scores below 4 - “bad to good” - or 5 - “bad to good” - and final IDEB scores above 6) and relative (those who remained in the last quartile of IDEB scores throughout the whole period were classified as at risk) performance metrics². However, different circumstances will lead to different choices on this specific point. Some circumstances may require the use of purely absolute or purely relative metrics.

The second step is to use this ranking to find neighboring schools that belong to opposite categories. Thus, for each underperforming school, the geographically closest benchmark school was identified. This step aims to reduce socioeconomic variations between the benchmark and at-risk schools, and seems to have been successful in the case of Rio (^{Ragazzo; Almeida, 2020}). This is an assumption that always needs to be verified. If the selection of neighboring schools is unable to ensure socioeconomic comparability, researchers should include new procedures to achieve that goal. For example, pairs with highly discrepant student bodies from a socioeconomic point of view should be discarded from the final set³.

The literature is mostly silent on the method used to select which schools will offer internships, which principals will be selected as mentors, and which principals should be encouraged to participate as students. Our method offers an answer to this question: within each pair of schools there is one potential principal candidate who should take on the role of mentee and another who can probably fulfill the role of mentor. In this context, physical proximity is also important to ensure that some internship or mentoring scheme between the two principals is pragmatically convenient. In contrast, in cities with a very low population density, or a low number of schools within each school system, this step may not reveal obvious opportunities for administrative intervention.

Given the location of the schools and the history of their performance, any school system can apply the method to find discrepant pairs. Our hypothesis is that where population density is high and the absolute size of the school system is large, this will usually reveal pairs that are physically close enough to enable effective mentoring programs between recipient schools and knowledge givers. This makes our procedure replicable and, in a sense, scalable, as the number of students potentially benefiting can be increased from repeating the procedure in different locations.

A regulatory instrument that seeks to exploit this method to increase the quality and equity of public education should begin by defining benchmark schools and at-risk schools (step 1). This classification, in turn, allows public administrators to identify the best pairs of schools to include in mentoring programs (step 2). Finally, under this instrument, administrators should have the ability to encourage principals from underperforming schools to participate in a mentoring program, as well as the ability to recruit principals from benchmark schools to act as mentors.

This would be a bottom-up regulatory instrument, since the specific curriculum of the training program should be left largely to the mentors. Within this proposal, there is no central body invested with the choice of which practices should be valued and encouraged and which practices should be abandoned. Of course, it is possible that after a few cycles, some general lessons could be learned and incorporated into other top-down regulatory instruments. But even in that case, the procedure ensures that local realities are able to inform public policy in a meaningful way.

In this section, we proposed that the pairs of schools identified by ^{Ragazzo and Almeida (2020)} can be used to create a mentoring program. This proposal, however, depends on the verification of an empirical premise. After all, several different factors may explain the discrepancy in results between pairs of schools. For example, it could be that these schools have very different teachers, or that the student bodies have significant differences not captured by the Indicator of Socioeconomic Status of Basic Education Schools (INSE). However, everything that has been said so far assumes that it is possible to verify whether the different results can be attributed, at least in part, to differences between the principals of the schools in each group. Is this the case in the city of Rio de Janeiro? In the next section, we seek to answer this question empirically, trying to find systematic differences between principals of schools that improved significantly and principals of schools that performed systematically badly.

Principals’ Characteristics and School Performance in Rio de Janeiro

As indicated in previous sections, principals have a broad influence on student educational outcomes. But what skills and characteristics make some principals more successful than others? The evidence reported on this important policy question is much less clear. The work of Ana Cristina Prado de Oliveira (^{Oliveira; Waldhelm, 2016}; ^{Oliveira; Carvalho, 2018}) showed that perceived principal leadership reliably predicts school performance in the city of Rio de Janeiro, suggesting that the key to quality education may lie with principals. Similarly, ^{Polon (2009)} showed that differences in leadership style can affect performance. However, leadership is an abstract concept, and perception is subjective.

While these characteristics are important and may help explain some of the variation in results among principals, we have chosen in this paper to explore an objective metric. Thus, we focus the analysis below on principal experience. This is one hypothesis - among many others - that may explain why some principals achieve better results than others, after controlling for socioeconomic variables that are known to have a strong impact: it may be that more experienced principals, because of the practical knowledge acquired through school management, create the pedagogical conditions necessary for student success.

Below, we report the results of quantitative analyses that aim to investigate the plausibility of this hypothesis (^{Osfhome, 2021}). It is important to note, however, that we do not claim to establish decisively the causal relationship expressed in the hypothesis. As we shall see, there are many interrelated factors involved, making inference of causality difficult. In any case, even if we are unable to establish causality, the repeated influence of variables indicative of experience in several different analyses is an indication of the plausibility of the hypothesis raised. To operationalize the research, we resorted to Prova Brasil (^{Brasil, 2021}), which also includes questionnaires aimed at students, teachers, and principals, yielding relevant information for this analysis. Previous work (^{Oliveira; Waldhelm, 2016}; ^{Oliveira; Carvalho, 2018}) has used items in the teachers’ questionnaire to assess principals’ leadership. The experience of principals can be extracted from the questionnaire completed by the principals themselves⁴.

We restrict our analysis to the early years of elementary school. This choice is justified, because deficiencies in the early years of basic education can compound to create more serious learning problems in later stages. Thus, if we need to prioritize according to the likely impact of our choices, the early years should receive more attention.

The simplest test of the hypothesis that principals’ experience is associated with higher IDEB scores is a bivariate correlational analysis. Using all 2,109 observations on early elementary school grades from municipal schools in the city of Rio de Janeiro between 2011 and 2017, we found a small but statistically significant Spearman correlation between grades and principals’ experience, as illustrated in Figure 1 ⁵.

Source: ^{Brasil (2021)}.

Figure 1 The performance of schools correlates significantly with the experience of their principals. Error bars represent the 95% confidence interval of the mean 

The results provide preliminary evidence for the idea that school performance is associated with principals’ experience. However, it is well known that factors related to the student body influence academic performance. In particular, socioeconomic status is usually noted as one of the best predictors of performance on standardized tests (^{Alves; Soares; Xavier, 2016}; ^{Alves; Soares, 2013}). Thus, a simple bivariate analysis is not sufficient to establish the effect we are interested in.

To address this, we built a hierarchical model with IDEB score as the dependent variable and principals’ experience as the independent fixed effects variable, including school and student body socioeconomic status⁶ as random effects variables. The model revealed that the effect of experience is significant, even considering schools with students from different socioeconomic backgrounds (p = .001)⁷.

There may also be a chicken and egg problem with our analysis. Perhaps, high-performing principals stay longer in their jobs and are therefore more experienced than those who perform poorly. After all, the selection of principals in the municipality of Rio de Janeiro is by election by the school community for 3-year terms (^{Rio de Janeiro, 2017}). Presumably, successful principals find it easier to retain their positions than principals who are less successful. Thus, the observed correlation between IDEB scores and experience could simply reflect this fact. In this paper, we are not concerned with establishing the casual direction of this relationship. More econometrically rigorous work that has looked at this question indicates that more experienced principals are better in part because of their experience (^{Clark; Martorell; Rockoff, 2009}), rather than remaining principals longer because of their superior skills. Even if this is not the case, however, the implications for public policy-the focus of this paper-would still be the same, as we argue at greater length below.

Thinking about the policy questions that motivate this investigation is also helpful in elucidating which metrics should be evaluated. Given that our goal is to reduce inequalities within school networks, we should not only focus on performance in the IDEB, but also - and crucially - on how this performance has evolved over time. If we want to make recommendations about improving underperforming schools, those schools that have consistently ranked among the best are not the appropriate paradigm. If we find that experienced principals are good at keeping high-performing schools at a high level but bad at improving low-performing schools, the policy implications are very different than those that follow if experience helps principals overcome obstacles associated with low-performing schools.

Focusing the analysis on schools that have emerged from historically low scores also has another virtue: given that prospective principals choose the schools they wish to lead, it may be that any differences between principals were caused by pre-existing differences between schools (e.g., some schools are better - because they have better infrastructure, are in a healthier community, have more committed faculty, etc. – and the best principals are simply targeting these schools as career goals). Given this explanation, the experience of the principal appears to have relatively little influence. In contrast, if we compare principals from schools that have improved with principals from schools that remain low-scoring, this explanation is not available. After all, at the first moment of analysis, both schools produced equally poor results.

Experience in the position as a proxy to find principals that could potentially be paired in training programs

Since we want to identify best practices for transforming schools with poor results into schools with good results – rather than simply noting which variables influence school performance – we should investigate schools that have followed this path. On the other hand, improvements in absolute terms are not enough to qualify a school as a benchmark. If, despite dramatic improvements in the school’s learning outcomes, the final quality identified is still low, the policy goal has not been achieved.

Taking these factors into account, we employed the same strategy described in ^{Ragazzo and Almeida (2020)} to classify schools between different levels of risk and benchmark. To be classified as a benchmark school, schools needed to have improved from a score below 4 (“very poor performance to good,” 24 schools) or 5 (“poor performance to good,” 124 schools) on the 2005 IDEB to a score above 6 on the 2017 IDEB. In contrast, to be classified as underperforming schools, the school had to have occupied the last quartile of the 2017 IDEB data and the last quartile of the average of the last 4 IDEBs (2011, 2013, 2015, and 2017). The idea behind this double standard is that we need to find regulatory solutions for schools that consistently have problems, rather than allocating resources to schools that may have performed poorly because of one-off problems that occurred in a specific year. One hundred and one schools with complete data in the database were identified using the two conditions.

Figure 2 shows that in 2013, 2015, and 2017, but not in 2011, schools in the benchmark groups had more experienced principals than underperforming schools. A direct comparison between the two groups (clustering the two benchmark strata) shows that the difference in principal experience is statistically significant for the full period (t(794.39) = 3.72, p < .001)⁸. However, an inspection of the patterns revealed by Figure 2 reveals that this difference is likely due to the dramatic reduction in the average experience of principals’ experience in underperforming schools between 2011 and 2015. While we can see a similar reduction in the experience of school principals in the benchmark groups from 2011 to 2013, there is no such difference between the 2013 and 2015 scores in these groups. This suggests that the schools’ underperformance may have been caused in part by higher turnover in principal positions relative to the benchmark schools. Moreover, this turnover appears to have replaced experienced principals with inexperienced principals.

Source: ^{Brasil (2021)}.

Figure 2 Schools that have improved significantly since 2005 have more experienced principals. Error bars represent the 95% confidence interval of the mean 

While the available data do not allow for a direct test of this hypothesis, a heuristic can help us estimate levels of turnover among principals. One of the metrics reported by principals in their questionnaire is the amount of time they have been in their current position as principal of that particular school. Whenever a principal reports a shorter time in office than that reported by the principal of the same school in the previous questionnaire, we can infer that there has been a change of principal. According to this estimate, underperforming schools changed principals significantly more often (mean = 1.13) than schools that went from performing poorly to well (mean = 0.79, t(173.18) = 4.74, p < 0.001) and then schools that went from performing poorly to well (mean = 0.84, t(1188.5) = 7.75, p < 0.001). In contrast, we observed no differences between the two benchmark school strata (t(178.09) = 0.69, p = 0.49).

While we do not claim to establish a causal relationship, many potential influences remain unmodeled in the above analyses. For example, Rio de Janeiro is a notoriously unequal city. Many cultural and socioeconomic factors vary geographically throughout the city. Thus, a cleaner test of the hypotheses drafted above about the role of principals involves identifying comparable pairs of schools to see if differences in the relevant dimensions still remain (^{Abrucio, 2010}).

^{Ragazzo and Almeida (2020)} identified pairs of schools that are physically close but belong to different levels of benchmark and risk in the typology discussed above. Each pair is composed of two schools: an underperforming school and the closest benchmark school (either a school that has moved from “poor performance to good” or “very poor performance to good”). The analyses reported in the paper point out that the resulting pairs are reasonably close (the smallest distance found is 98 meters, and only five pairs are separated by more than three kilometers) and that the differences in performance between the two schools in each pair cannot be reduced to differences in the socioeconomic status of their student bodies.

We used the pairs identified by ^{Ragazzo and Almeida (2020)} to investigate our hypothesis regarding the influence of principals’ experience on educational outcomes in the IDEB. For each pair and each question in the principals’ questionnaire, we computed the difference between the answers given by principals from the benchmark school and the underperforming school.

Finally, we averaged the differences to find the questions that were answered most differently by the two classes of principals. In ranking the 111 items from the principals’ questionnaire, we saw that principal experience (third item with least agreement, 15.84%), time in office (lowest agreement, 14.85%), reported problems with financial resources (fifteenth item with least agreement, 28.71%), and teachers suitable for subjects (sixteenth item with least agreement, 28.71%) occupy some of the highest positions⁹. Replicating the analysis done earlier, considering only pairs of data (which may cause a benchmark school to be counted multiple times, as it may be the closest benchmark to multiple underperforming schools), we again detect a significant difference in principals’ experience, such that principals of benchmark schools are more experienced than those in charge of at-risk schools (t(197.75) = 3.54, p < 0.001). Taken together, the above analyses seem in line with the literature in asserting that school principals matter (^{Branch; Rishkin; Hanushek, 2013}) and that experience, in particular, is especially important for their performance, as argued by ^{Clark; Martorelle; Rockoff (2009)}.

The best way to explore the importance of the principal in achieving public policy goals is to explain the patterns of results observed above. The hypothesis that more experienced principals achieve better results because of the administrative and pedagogical lessons they have learned over their careers is consistent with the data. If we identify less experienced principals in comparable schools, it is possible that the lessons learned in practice by more experienced principals could be passed on more quickly through public policies that facilitate this exchange. More specifically, we believe that a training program for school principals with a mentoring component from the pairs of schools identified above would work as an effective strategy to reduce existing inequalities within the school system of the municipality of Rio de Janeiro.

Conclusion

While the literature is consistent in pointing to the importance of school leadership on academic performance, there is less clarity about the precise mechanisms that distinguish some principals from others. This paper contributes to this literature by showing that, at least in Rio de Janeiro, experience appears to be one such mechanism. Moreover, the methods we employ to demonstrate the influence of experience can serve as the basis for a bottom-up regulatory strategy involving the training of principals. By using data to identify pairs of physically close schools with student bodies of similar social and economic backgrounds, we can select the right actors to play the roles of mentor and mentee in training programs that can help with the specific challenges each school faces. This method is scalable, replicable, and inexpensive. School systems with a high number of schools in highly populated areas could benefit from adopting this strategy.

Our method is also replicable and scalable in a different sense: at any point in time, we can reapply the method to find new relevant pairs of principals. Suppose that, at a given moment, the city of Rio decides to implement a public policy along the lines advocated here. Four years later, the results are great: the schools that originally underperformed no longer occupy the last quartile of IDEB scores, while benchmark schools still perform adequately. Instead of closing the program and shifting the focus to new types of educational policy, a possible way forward would be to repeat the procedure. Since the first cycle was successful, the application of the method should reveal new (and better) underperforming schools, which would now be those with more to gain from a mentoring program.

This strategy also implies low costs. It does not require cities to pay for expensive instructors, nor does it involve the creation of costly physical infrastructure. It relies only on inexpensive data analysis (especially in places like Rio, where data is already collected and publicly available for free) and on the existing workforce, which can be incentivized at much lower costs than would be required to obtain outside help. The assumption is that more experienced principals are better not only because of their seniority, but also because they acquire specific skills and knowledge applicable to their school context. If this assumption proves true, we should expect that these skills and knowledge would also be applicable to neighboring schools serving a similar socio-economic population.

We emphasize that this proposal is agnostic about the causal structure of the observed effect. If experience improves principals, we should leverage their experience in training programs. On the other hand, if the effects are caused by selection (only principals with good results last long enough in their positions to gain experience), we should still select them to train novice principals. Thus, the data support the proposed public policy even if our assumptions about causality are wrong.

Finally, although we believe that principal experience is associated with school performance, our method can still be useful in contexts where this is not true. Other characteristics of principals that vary systematically between high and low performing schools can also be identified and addressed from the proposed method. Imagine, for example, that some pedagogical technology that is used correctly by only a few principals is the reason for a gap. Presumably, by matching principals from high- and low-performing schools, we would also discover this problem and could act to remedy it.