Context evaluation: Around the instruments

The concept of context that is used in ECE is very important to define the evaluation types at this stage of basic education. Therefore, we highlight the conception presented by ^{Bondioli (2014)}, used in national documents (MEC, 2009), according to which context consists of concrete elements (people, furniture, materials) and symbolic relationships, in a dynamic and reciprocal way. The author argues that coherently choosing a self-evaluation instrument should take into account the reality to be evaluated and the principle of negotiation; she also emphasizes that the evaluation process should be based on the pedagogical approach. Her perspective builds on an ECE approach developed in Italy, and it is worth emphasizing that the author refers to the unit’s environment, characteristics, processes, and project.

However, the ECE context comprises several levels: the policies, which define the service modes (Lei n. 9.394, 1996); the norms on initial teacher education, which define the theoretical-practical framework of undergraduate pedagogy and licensure programs; the continuing education of teaching staff, the Base Nacional Comum para a Formação Continuada [National Common Teacher Education Base] (BNC - Formação Continuada) (Resolução n. 2, 2019), which also contains provisions on the Diretrizes Curriculares Nacionais para a Formação Continuada de Professores da Educação Básica [National Curriculum Guidelines on Continuing Teacher Education for Basic Education Teachers] (Resolução CNE/CP n. 1, 2020); the Diretrizes Curriculares [Curriculum Guidelines] (Resolução n. 5, 2009), which set the principles and conceptions underpinning the pedagogical work with children; and policy implementation in organizing institutional teams, as well as units’ pedagogical work, classrooms, and children, according to the Base Nacional Comum Curricular [National Common Curriculum Base] (BNCC) (MEC, 2017). This set of documents delimits the offer conditions, that is, the ECE context is defined by this chain which involves legislation, guidelines, infrastructure, professional training, and pedagogical plans. EI context evaluation should encompass all these aspects, monitoring policies and the quality of practices.

From this perspective, the literature on educational evaluation considers three important aspects for ECE: children’s access to vacancies; the inputs that characterize the supply conditions, which in turn imply several levels of evaluation (funding, human resources, facilities infrastructure, material and pedagogical resources, and compliance with legislation); and processes, encompassing the way in which adults and children are organized for the pedagogical process, i.e., how the educational plan is implemented. From the perspective of educational evaluation, context includes aspects that structure the policy and pedagogical practices carried out in the units. Of these, it is common for processes to be the most controversial ECE dimension, not only because of their theoretical-practical complexity, but also because of the research methodology that is chosen. As an evaluation focuses on professionals’ and children’s actions, it may take on different emphases depending on the evaluation and monitoring approaches, which can be complementary and participatory. In addressing the issues related to university reforms and the plans for participatory institutional evaluation, ^{Leite (2005)} sheds light on the discussion on educational evaluation types, arguing that processes: (1) require awareness, agreement, negotiation, and persuasion; (2) are epistemological instruments for thought and political action in the public sphere; (3) include the creation of interpretive communities guided by the common good (a sense of belonging), with actors who view themselves from a relational perspective, in the interplay between knowledge, powers, and duties; (4) are exercised through co-management and self-government; (5) institute autonomy for individuals, collectives, and the organization; (6) are nurtured by permanent self-legislation, self-criticism, and self-monitoring; (7) require their own time. The combination of processes requires a fresh look at external large-scale evaluation, which seems to involve actions that are disconnected from the public to be evaluated, and efforts must be made towards understanding what the evaluations together can produce for all those involved.

Table 1 presents some characteristics of the various forms of evaluation and their focuses, considering the principles of participatory institutional evaluation of ECE and collective construction of the evaluation policy in question.

^{Sordi (2022)}, when presenting the Participatory Institutional Evaluation model for the municipality of Campinas, emphasizes that it is supported by quality principles negotiated between schools and public authorities and by the supportive participation of all segments involved with the school unit. Mendes et al. (2015, as cited in ^{Sordi, 2022}, p. 32, own translation) also explain that “the advocated participation consists in promoting dialogue between the actors at various system levels, considering paid pedagogical time, spaces, and autonomy (both pedagogical and financial)”. The evaluation policy for primary education and youth and adults education (EJA, in Portuguese) at the municipal level (Resolução n. 5, 2009) establishes an internal evaluation committee (CPA, in Portuguese), whose tasks include dialogue with external evaluation processes. In the field of ECE, ^{Bondioli (2014)} proposes something similar regarding the definition of the concept and principles of quality.

With regard to methodology, Table 1 presents the various evaluation possibilities and scope and shows that it can be seen as a set of institutional evaluation strategies founded on democratic and participatory principles. The participatory institutional evaluation carried out by the teams from the educational unit should include both self-evaluation and child evaluation processes, usually called the Relatório de acompanhamento da criança [Child monitoring report], in compliance with the legislation that determines this as a task of teaching staff (MEC, 2009). System evaluation, a possible form of participatory institutional evaluation, refers to evaluations that seek data to inform policy, practice, and continuing education actions and may also address access, input, and process issues. Child evaluation refers to studies and research that provide insights into the trajectory of each child in terms of their development and learning, not necessarily as a child performance evaluation policy.

While instruments should be chosen based on discussions with school segments, it is important to analyze evaluation instruments’ ability to measure staff perceptions and, on the other hand, educational processes. An instrument intended to evaluate the various aspects of the ECE context should follow design, testing, and validation steps in a scientific and systematic way (^{Pasquali, 2013}), which gives the instrument legitimacy to evaluate phenomena such as pedagogical practices and staff perceptions about their own work. Thus, there is a difference between validated instruments intended to measure phenomena through quantitative evaluation research and instruments built for specific self-evaluation purposes, such as those developed to discuss and monitor the implementation of an education unit’s pedagogical plan, with the school community participating in discussion and reflection processes. Any context evaluation instruments, be they psychometrically validated or not, can be used in professional training actions.

Thus, Table 2 presents the scope of the instruments that are available in Brazil.

During the process experienced with members of the school community, self-evaluation instruments help school community members reflect on their actions and planning; additionally, they allow including topics related to what is proposed in the instruments. Because results for the national and São Paulo indicators are recorded by the participants through a color system (green, yellow, and red), it is possible to conduct studies describing the self-evaluation patterns. However, the descriptors/indicators forming these instruments’ dimensions are often generic, but at times they can be more focused and objective. Still, in both cases, when used across many school units, they may lead teams to read, interpret, and understand the indicators differently, driving a view focused only on their own reality.

The teams’ reflection and discussion processes may address aspects that escape the scope of analyses focused on describing a particular situation based on variables. For example, indicator 5.1 of the Indicadores da qualidade na educação infantil [Early childhood education quality indicators] is described as follows: “spaces and furniture that promote children’s experiences” (MEC, 2009, p. 33, own translation). “Spaces and furniture” can vary greatly between units, and whether they “promote children’s experiences” is something that should be evaluated based on the unit’s specific initiative. Therefore, it is difficult to pinpoint the characteristics of the spaces and furniture that the indicator is referring to; and the phrase “promote children’s experiences” can be interpreted in various ways. Despite this limitation, the information is useful for participants in the self-evaluation, reflection, and analysis process, helping to direct attention to necessary improvements, which shows the instrument’s formative nature.

On the other hand, instruments designed for the external evaluation of different contexts require greater objectivity and focus both in their design and application processes. User training should be organized to ensure they fully understand the concepts used in its composition and know how to seek the necessary information. Besides ensuring the study and thorough understanding of dimensions, items, and descriptors, evaluator reliability must also be checked, that is, whether evaluators mark the observed phenomenon coherently. This ensures that the instrument is being used correctly and demonstrates that it is well-calibrated and conceptually sound, and that the results are reliable, especially when used to evaluate large sample groups by a large number of field researchers.

This being the case - the use of foreign instruments selected for the evaluation procedure - an adaptation should involve psychometric procedures. Importantly, using only parts of the instruments should be avoided to maintain internal consistency, that is, the extent to which the total scale and the subscales measure common concepts (^{Harms, Clifford & Cryer, 2020}, p. 11). In addition, an instrument should not be translated or adapted to specific realities without its authors’ permission.

Considering instruments’ different scope and goals, they should not in principle be compared. However, a qualitative content analysis can be conducted, starting from each instrument’s underpinning concepts towards identifying the theoretical decisions made to address similar phenomena. ^{Guerres-Zucco et al. (2022)} present an analysis of eleven instruments, of which only five are published or freely accessible in Portuguese; it provides an overview of some of their specificities and scope. However, the authors do not mention that instruments such as ITERS-R and its version 3 and ECERS-R and its version 3, or the Australian instrument (NQS),³ have been used in self-evaluation processes as well. The two versions of ITERS and ECERS (version R and version 3) are being used, at their authors’ and users’ suggestion, by educational units in self-evaluation processes, in preparation for and to understand external evaluations, and in initial and continuing education actions, without restrictions for acquisition or use. Material containing training instructions is also available.

The Australian instrument is available to all units in that country, whether private or public, for internal use. Its intended use includes debates between evaluators and units on the external and self-evaluation results (^{Bhering et al., 2020}). This dialogue has influenced not only the instrument review processes over the years but has also fostered discussions between the central evaluation body - Australian Children’s Education and Care Quality Authority (ACECQA) (2020) - and the education units. In addition, the instrument offers opportunities for internal discussion on educational and pedagogical practices (^{Bhering et al., 2020}). Institution teams, through the results of their self-evaluation, have the right to question the results of the external evaluation by requesting reviews of their scores and measured quality level. It is worth noting that this instrument is used to evaluate all units in the country, which means that it was built with sensitivity to include all units that adopt the national curriculum or are inspired by educational approaches such as High-Scope, Waldorf, Montessori, and Italian approaches, project methodology, etc.

Instruments that generate quantitative data can be used in any evaluation format, while self-evaluation instruments should be carefully analyzed in psychometric terms before deciding to use them to generate results that can be generalized. This difference is crucial because thorough study and careful analysis can aid in understanding the tool construction and applicability processes, as well as conceptually understanding the educational phenomena that are considered critical for ECE quality and for instrument composition.

Table 3 presents the structure of the instruments that are accessible in Brazil and points out, through different colors, their similarities in terms of their dimensions.

The Italian instruments are structured differently. ISQUEN - created by Becchi et al. in 1999 - consists of 51 descriptors distributed across four areas of interest: 1) subjects; 2) contexts and practices; 3) knowledge from doing; and 4) guarantees. Each of these areas has subareas. AVSI - created in 2001 by ^{Bondioli and Ferrari (2008)} - comprises 91 descriptors organized into four areas of interest: 1) educational experience; 2) professional activities; 3) adults and their relationships; and 4) guarantees. Each instrument uses a different scoring system. Each item of the ISQUEN scale has three non-exclusive conditions, and observation can determine which relate to the evaluated context. AVSI is constituted as an item-based instrument tiered into nine levels of quality, with descriptions in five of the nine levels (1, 3, 5, 7, 9), and scores can pertain to that sequence or to the intermediate one (2, 4, 6, 8) (^{Bhering et al., 2021}).

The ECE teacher questionnaire from the Saeb-EI consists of the following dimensions: general information, training, professional experience and working conditions, class and classroom characterization, and teaching materials and resources (toys, books, and arts, mathematics, and music materials). Items in the pedagogical resources dimension are evaluated on a four-point Likert scale, and some questions address the processes dimension of the evaluation tripod, referring to children’s autonomy when using the resources, especially books. The principals’ questionnaire has the following dimensions: school general characterization, personal information, and working conditions, resources, and infrastructure, as well as management and participation. A predominance of the input dimension is seen in both instruments.

Brazilian research results: ECE context evaluation

In 2008, the MEC organized a research effort to understand the state of Brazilian ECE after 12 years of the Lei de Diretrizes e Bases da Educação Nacional [National Education Guidelines and Framework Law] (LDB). This effort sought information about the services provided by public, private, and voluntary educational units in six Brazilian capitals and examined the impact of kindergarten on the results of Provinha Brasil on the performance of children aged 7 to 8 years. The Fundação Carlos Chagas [Carlos Chagas Foundation] (FCC) was selected and partnered with the MEC and the Inter-American Development Bank (BID, in Portuguese) to develop the study Educação Infantil no Brasil: Avaliação qualitativa e quantitativa [EI in Brazil: A qualitative and quantitative evaluation]. The sample defined in the bid notice comprised: Belém (PA), Fortaleza (CE), Teresina (PI), Campo Grande (MS), Rio de Janeiro (RJ), and Florianópolis (SC). One of the various challenges facing the study was choosing the quality evaluation instrument, considering the Brazilian reality.

The team sought national instruments that were available, validated, and reliable. However, there were no Brazilian instruments at the time that met the required criteria, so international instruments were then sought. The FCC research team analyzed Brazilian literature and documents that might inform the analysis of the instruments to be considered. ^{Campos et al. (2010}, p. 55, own translation) indicated that:

The processes of choosing and adapting the observation scripts required special attention from the team, as they needed to meet several criteria: being consistent with the national official documents, containing curriculum guidelines and quality parameters for daycare centers and kindergartens; being sufficiently detailed to allow a service quality evaluation that might cover a wide range of aspects, including the main dimensions of early childhood education quality identified in the literature; and allowing adaptation to the research goals within the given schedule.

Among the analyzed instruments, ITERS-R and ECERS-R were included because their dimensions covered important aspects of Brazilian ECE; they had a play and interaction-based approach; they were used in many countries, which certified their reliability, validity, and sensitivity to cultural variations; theoretical and practical training was provided for use in large-scale research; and their translations had been tested and studied in Brazil. While scientifically founded, this choice caused discomfort in the ECE field, as this was a new research initiative in the country. There were concerns not only that results would not actually portray the Brazilian scenario and might drive an evaluation not consistent with the national plans, but also that the generated information might be misused. The analyses did not confirm the non-representation concerns and contributed to new debates on the possible paths and positions that would guide researchers and education professionals in Brazil.

Subsequent research confirms the need to deepen knowledge regarding ECE context quality at the municipal level, as the lack of data did not allow sufficient understanding of the service. In addition, it was not possible to determine whether services met the Parâmetros nacionais de qualidade para a educação infantil [Early childhood education national quality parameters] (MEC, 2006) and whether children’s fundamental rights were being guaranteed according to Brazilian legislation. The initiative strongly influenced the course of this discussion. Important positions defined the boundaries of debate, the form and content of procedures that could potentially impact the full development of service for children aged 0 to 5 years in the first stage of basic education (MEC, 2012, 2015).

Studies on ECE context quality have been developed over the last 13 years. They all provided feedback to the municipal systems involved, presenting their results and enabling debates. In addition, the research and evaluations included members of research groups, academics, and municipal department staff who conducted the external evaluation of the educational units. This partnership allowed a rich and powerful dialogue between teams on evaluation, quality, improvements, and forms of analyzing and using results. Table 4 presents the studies’ information about its instruments, sample and reports.

MEC/BID/FCC study

The FMCSV/LEPES-USP/RP study

Final considerations

The instruments selected to gather information about ECE, considering scientific methods, should align with the goals of the evaluation, i.e., this choice should be based on the instrument’s ability to respond to or achieve the pre-established goals. Evaluating ECE in representative samples, e.g., by investigating how the institutional contexts and their resources (inputs) are organized, or the pedagogical practice in relation to processes, presupposes selecting or designing instruments with careful consideration of the stages of their constitution and testing procedure. In other words, whether they are tested for reliability and validation. In any case, instruments for educational evaluation should be tested, unless the evaluation process is conducted reflexively, and over the process they may be changed and adjusted in order to capture participants’ positions more reliably.

No Brazilian instruments for evaluating ECE contexts have been published, except for the Indicadores da qualidade na educação infantil (MEC, 2009), whose methodology presupposes a collective discussion and reflection process based on statements regarding various aspects of institutional action. As said earlier, foreign instruments have been translated and published that can be applied for external evaluation and self-evaluation. Additionally, there are instruments inspired by the Indicadores Nacionais [National Indicators] and designed by municipal education departments for reflective processes.

Brazilian scientific production in the field of ECE context evaluation discusses the adequacy of foreign instruments for evaluating national contexts, the ways of conducting evaluations, and how results are used. In this respect, there is a misunderstanding about the contribution of external evaluations as regards the improvement of ECE contexts and especially as concerns teacher education processes. It is usually considered that external evaluations not only maintain a distant perspective on the complexity of ECE service, but also fail to promote a broad understanding of their results, making it more difficult to engage those involved. Thus, methods related to self-evaluation are advocated and preferred as they include the participation of school communities and consider the opinion of all members. Along the same vein is a justified defense of the importance of ECE in the lives of children and their families and, therefore, a struggle to maintain quality service (especially in day care centers).

Therefore, it is necessary to clarify the role of evaluations without losing sight of public policy goals and the need to obtain reliable information about ECE service. It is necessary to stress the importance of the various ways of looking at the same phenomenon and investigating the various aspects involved in performing quality work, thus guaranteeing what is unnegotiable. It is essential to examine whether what is contained in the mandating documents, built by various hands and discussed in various forums, is actually reaching all children according to national expectations. Therefore, going beyond the theoretical debate about the ECE project means getting to the day care center and kindergarten floor in a scientific, ethical, and formative way in order to ensure quality education, providing learning for all children, in the national scenario. Finally, it is also necessary to clarify that, despite their informative potential, evaluations should not - in a country like Brazil, where inequality persists - be at the service of punitive or meritocratic measures and should not be carried out in a hierarchical manner (^{Sordi, 2022}). On the contrary, evaluations must be at the service of pedagogical work evolution, staff training, structural change, the necessary adjustments to public service policy, and above all, the welfare of children, ensuring them their inalienable rights.

Evaluations support the development of educational and social service, strengthen the dialogue between the different public education levels (^{Sordi, 2022}), and enable decision-making that takes into account all those involved. They should be questioned, reviewed, updated and, above all, informed about how their audience perceives their implementation, conduct, results, and directions. It is of interest that their audience understands them so that it incorporates them in a formative way into the daily dynamics of ECE work, with the sole purpose of organizing intentional environments towards the full and meaningful learning of Brazilian children. It is desirable that the different types of evaluation inform the various levels of public policy and have an impact on educational units’ daily practices. It is necessary to know ECE at the national, state, and municipal level, so that measures are taken to improve quality based on the various Brazilian scenarios.

The step forward in approaching ECE context evaluation would be to collectively define how the municipal ECE plan can be evaluated in its political, theoretical, and practical dimensions, how the instruments should be constituted for evaluations, who should lead the self-evaluation and large-scale external evaluation processes, and how the results will be analyzed and put into perspective by units, both individually and collectively. Thinking about the school should be done from various perspectives, from different places, such as the school community, municipal systems, and researchers. According to Bondioli’s (2004) arguments, this attitude reaffirms the negotiated quality pact, without excluding what has become non-negotiable for the country.