INTRODUCTION

PISA (Program for International Student Assessment), developed and coordinated by the Organization for Economic Cooperation and Development (OECD), is an international triennial assessment that, in addition to reading and mathematics, also focuses on science. Student performance in PISA is measured through tests, and in each assessment cycle, one of those three cognitive areas is the main focus, with most items centered on that area (approximately two thirds of the total test).

PISA assesses students with ages between 15 years and three months and 16 years and two months by the beginning of the test administration period, an age bracket that presupposes the end of compulsory basic education in most countries, and also that students are at least in 7^th grade. The Program is used as an assessment tool in many regions around the world. It was implemented in 43 countries in its first edition (2000), 41 in the second (2003), 57 in the third (2006), 75 in the fourth assessment (2009), 65 in the fifth (2012), 72 in the next-to-latest (2015) and 79 in the latest edition, held in 2018.

In this study, we used data from PISA 2006 and 2015, which highlight scientific literacy, thus allowing to measure students’ performance regarding the competencies and knowledge assessed in specific contexts. The vision of scientific literacy at the basis of PISA can be summarized in the following question: what is important for young people to know, value and be able to do in situations involving science and technology? (ORGANIZATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT - OECD, 2006, 2008, 2013a, 2016).

This study’s main goal is to understand how scientifically prepared the Brazilian students who participated in PISA 2006 and 2015 are. To that end, we seek to answer the following research questions: how are Brazilian students situated in the international context in terms of scientific literacy in PISA? What grade are they in? How do Brazilian normal-age and over-age students do in the competencies and areas assessed by the Program? In order to continue the discussion and answer these questions, this article is organized into five sections, in addition to this brief introduction. Next, we approach the definition of scientific literacy used in PISA since its first edition, then we describe our methodological approach. Based on that, the results are presented and discussed and, finally, we conclude with our final considerations. It is worth stressing that, given the nature of the study, the discussion of methodological aspects will also pervade the subsections of results.

SCIENTIFIC LITERACY IN PISA

In both PISA 2000 and 2003 editions, which focused, respectively, on reading and mathematics, scientific literacy was defined in the same way:

Scientific Literacy is the capacity to use scientific knowledge, to identify questions and to draw evidence-based conclusions in order to understand and help make decisions about the natural world and the changes made to it through human activity.” (OECD, 1999, p. 60; OECD, 2003, p. 133)

In both 2000 and 2003, the definition incorporated knowledge of science and understanding of science within the term “scientific knowledge”. The definition in PISA 2006 separated and elaborated this term by splitting it into two components: “knowledge of science” and “knowledge about science” (OECD, 2006). Both definitions, however, refer to the application of scientific knowledge to understand and make decisions about the natural world.

To the original definition, PISA 2006 added knowledge of the relationship between science and technology - an aspect that was assumed, but not elaborated, in the definitions of 2000 and 2003 (OECD, 2013a). In that edition, an individual’s science literacy is defined in terms of:

The competencies evaluated in PISA 2006 are broad and include aspects related to personal utility, social responsibility and the intrinsic and extrinsic value of scientific knowledge (OECD, 2006). Thus, the Program’s perspective differs from those exclusively grounded on the curriculum and discipline of Science, but includes problems situated in educational and also professional contexts, thereby recognizing the essential place of the knowledge, methods, attitudes and values ​​that define scientific disciplines (OECD, 2006 ). The term ‘scientific literacy’ was chosen by Pisa, according to the OECD (2006), because it represents the goals of science education that should be applied to all students. This means a breadth and nature applied to the purposes of scientific education, representing a continuum of scientific knowledge and cognitive skills associated with scientific research, which incorporates multiple dimensions and includes the relationship between science and technology. Together, the scientific competencies at the center of the definition characterize the basis of scientific literacy and the goal of PISA 2006 scientific evaluation - i.e., to assess the extent to which the competencies are developed (^{BYBEE, 1997a}, 1997b; ^{FENSHAM, 2000}; ^{LAW, 2002}; ^{MAYER; KUMANO, 2002}; ^{GRABER; BOLTE, 1997}; MAYER, 2002; ^{ROBERTS, 1983}; UNITED NATIONS EDUCATIONAL, SCIENTIFIC AND CULTURAL ORGANIZATION - UNESCO, 1993).

Unsurprisingly, the scientific literacy domain’s definition did not changed in the 2009 and 2012 editions, since in those editions the focus was, respectively, on reading and mathematics. The ideas expressed in the definition are incorporated in the Program’s penultimate edition, in 2015, which again focused on science. The main difference is that the notion of “knowledge about science” was more clearly specified and divided into two components - procedural knowledge and epistemological knowledge (OECD, 2013a, 2016). Procedural knowledge is based on recognizing and identifying the traits that characterize scientific research, i.e., it requires knowledge of the standard procedures underlying the various methods and practices used to establish scientific knowledge. Epistemological knowledge, on the other hand, is an understanding of the logic of practices common to scientific investigation, the status of the generated knowledge’s claims and the meaning of essential terms such as theory, hypothesis and data.

According to the OECD (2016), people need the three forms of scientific knowledge - content, procedural and epistemological knowledge - to apply the scientific literacy competencies. Therefore, PISA 2015 sought to assess the extent to which 15-year-olds are capable of displaying those competencies properly within a range of personal, local, national and global contexts. Similarly to the 2006 edition, this perspective differs from that of many school science programs, which are often dominated by content knowledge. Rather, the Program’s matrix is ​​based on a broader view of the type of scientific knowledge that would be required of active members of contemporary society. In sum, the 2015 definition builds on the 2006 definition.

Scientific literacy is the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen. A scientifically literate person is willing to engage in reasoned discourse about science and technology, which requires the competencies to:

1. Explain phenomena scientifically - recognize, offer and evaluate explanations for a range of natural and technological phenomena.

2. Evaluate and design scientific enquiry - describe and appraise scientific investigations and propose ways of addressing questions scientifically

3. Interpret data and evidence scientifically - analyze and evaluate data, claims and arguments, and draw appropriate scientific conclusions.² (BRASIL, 2016, p. 37, our translation)

According to PISA’s report on data from 2015 (BRASIL, 2016), becoming scientifically literate involves the idea that the purposes of scientific education must be broad and applied; therefore, the concept of scientific literacy refers to knowledge of both science and science-based technology. These two areas differ as to their purposes, processes and products: while technology aims at optimal solutions to human problems, science seeks the answer to specific questions about the natural world. However, both are closely related.

Scientific literacy requires not only knowledge of science concepts and theories, but also knowledge of the common procedures and practices associated with scientific research and how they enable the advancement of science. According to PISA (OECD, 2016), scientifically literate individuals have the knowledge of the major concepts and ideas that form the foundation of scientific and technological thought and how such knowledge is obtained and justified by theoretical evidence or explanations. Therefore, in 2015, scientific literacy is defined in terms of the ability to use knowledge and information in an interactive way.

In sum, PISA, as well as the most recent contributions in the pursuit of a consensual definition of scientific literacy, combines the two major domains centered on an understanding of both the scientific content and the social function of science and technology, including attitudes, beliefs and interests that influence decisions and actions from a personal, social and cultural perspective. An understanding of science and technology is essential in a young person’s education for life in modern society. By expanding its interests in the field of educational assessment, and in order to compare Brazilian results with those of other countries, Brazil has been voluntarily participating in PISA since its first edition. Next, we present our methodological approach, and then we begin the presentation and discussion of this study’s results.

METHODOLOGICAL APPROACH

To answer the questions of this study, we conducted an exploratory analysis of Brazil’s results in the two editions of PISA that focused on science: 2006 and 2015. The results of Brazilian students were also explored and compared with the average of OECD member countries.

The analysis of results uses descriptive statistics and comprises univariate and bivariate analyzes. The importance of this type of analysis lies in the fact that no statistical data modeling can do without an exploratory analysis that allows the researcher to know the variables’ behavior (^{BABBIE, 1999}). The variables used³ in this first part of the study were selected based on the PISA student database available on the OECD website.⁴

The variables correspond to our units of analysis of results. We compared Brazil and OECD in terms of the general average in science, the levels of proficiency, the competencies in science and the areas of knowledge in science. For the performance between countries, we also chose to compare over-age, normal-age and advanced students in Brazil. This option is due to the fact that age-grade distortion is one of the dimensions of education inequalities and, in addition to performance, one of the main issues addressed by federal, state and municipal public policies in the country. Moreover, the specialist literature shows that grade repetition is not the best solution to teaching-learning problems in Brazil, and it is also a negative measure for performance in science in PISA, mainly because these students would have lacked the opportunity to learn what the Program typically evaluates among 15-year-olds. Given the problems of student flow in Brazil, the general averages in PISA were analyzed in four dimensions: general (all students taking the test); over-age students (those enrolled in primary education); normal-age students (those enrolled in the first grade of secondary education); and advanced students (those enrolled in the second and third grades of secondary education).

These dimensions were obtained by recoding the variables “st01q01 - grade” and “st001d01t - student international grade” from the 2006 and 2015 PISA student questionnaires, respectively, with the creation of a new variable categorized as “over-age”, “normal-age” and “advanced”, which allowed observing Brazilian students’ results in each situation. Our discussion of the study’s results is based on the literature on the negative effects of grade repetition as a pedagogical strategy to face teaching-learning problems in Brazil (^{FREITAS, 1947}; ^{BRANDÃO; BAETA; ROCHA, 1983}; ^{COSTA-RIBEIRO, 1991}; ^{ALVES; ORTIGÃO; FRANCO, 2007}; ^{CRAHAY, 2002}; ^{CORREA; BONAMINO; SOARES, 2014}). We also established a dialogue with previous findings of studies which analyze Brazilian students’ performance in PISA and, mainly, discuss the relationship between grade repetition and student performance in the Program (DALTON, 2011; ^{IKEDA; GARCÍA, 2014}; ^{GARCÍA-PÉREZ et al., 2014}; ^{MATOS; FERRÃO, 2016}; FERRÃO et al., 2017; ^{DIRIS, 2017}; ^{SASSAKI et al., 2018}; ^{ALVARADO et al., 2018}), and we used a linear regression model to investigate the effect of the grade repetition variable on Brazilian students’ performance in science in PISA.

Next, the study’s results are presented and discussed. First, we deal with eligible students and their ongoing school grade by the time they participated in the two editions of the Program considered herein. Then, we focus on Brazilian students’ average score in science, levels of proficiency, competencies and, finally, areas of knowledge in the 2006 and 2015 editions of PISA.

AGE-GRADE DISTORTION

It was not until the 1990’s that Brazil completed the process that led to universal basic education in the country. Alongside recent real progress, both in the multiplication of and access to educational offer, a retrospective look reveals the discriminatory character of the development of mass schooling in Brazil. High grade repetition rates, for example, are a problem that has long plagued Brazilian education (^{FREITAS, 1947}; ^{BRANDÃO; BAETA; ROCHA, 1983}; ^{COSTA-RIBEIRO, 1991}; ^{ALVES; ORTIGÃO; FRANCO, 2007}; ^{CORREA; BONAMINO; SOARES, 2014}). While this problem has decreased in Brazil, especially during part of the 1990’s, non-progression (repetition and dropping out) has persisted with extremely high rates.

PISA has been proving to be a good tool for studying aspects related to grade repetition, and several authors have been discussing this issue based on the assessment’s results. We can see, however, that such studies point to three perspectives. The first correlates repetition with student and school demographic characteristics. In this respect, it is worth highlighting the contributions of ^{Matos and Ferrão (2016}), for example, which analyze the phenomenon of grade repetition in Brazil and Portugal, using data from the 2012 edition and seeking to identify student and school characteristics associated with the probability of grade repetition in order to estimate variability between schools. Likewise, Ferrão et al. (2017), also based on data from 2012, evaluate the effects of both the school’s socioeconomic composition and the proportion of repeat students on the probability of grade repetition.

The second perspective concerns studies that analyze the determinants of grade repetition in education levels and its impact on student performance (^{GOOS et al., 2012}; ^{PEREIRA; REIS, 2014}; ^{DIRIS, 2017}). For example, ^{Ikeda and García (2014}) used the PISA 2009 data set and a sample of 30 countries and economies with more than 10% of 15-year-old students who had repeated a grade at least once to analyze the relationships between grade repetition and reading performance. The authors divided students according to the education level (primary or secondary) in which they had repeated a grade. The results show that this distinction between students who repeated grades in primary or secondary education is important, since the extent of the relationship between grade repetition and academic achievement differs depending on whether the students repeated a grade in one or the other education level. In most of the surveyed countries, students who repeated a grade in secondary education tend to outperform those who repeated a grade in primary education, and non-repeat students tend to perform even better than those who repeated in secondary education.

The third perspective pervades the previous two and analyzes the impacts of grade repetition on PISA results. Dalton (2011), for example, examines how the age-grade distribution and the grade per se of 15-year-old students contribute to performance in science in 27 countries participating in PISA 2006. The author notes wide variations in student distribution over the assessed school grades and finds that the cost of being in a lower grade than that of peers of the same age is higher in certain countries such as Greece and Spain. Conversely, the benefits of being in a more advanced grade are greater in other countries such as Australia and Luxembourg. In sum, even controlling his analyzes for demographic characteristics at the student and school level, Dalton (2011) shows that being an over-age or an advanced student can heavily influence performance in science in many OECD countries. ^{Alvarado et al. (2018}) evaluate the impact of grade repetition on Colombian students’ performance in reading, mathematics and science in PISA 2015. The authors’ results, like those of the studies above, also show that students who were not retained have average scores above that of their country in PISA. Using a t-test, the authors found significant differences between the average scores of Colombian repeat and non-repeat students in the three domains assessed by the Program.

The results of PISA 2009 underscored Brazil’s situation by revealing that 40% of Brazilian students repeat at least once during basic education (OECD, 2010). PISA reports released more recently suggest that repetition is an expensive policy, which is sometimes used as a form of punishment to sanction misbehavior at school, and may reinforce education inequalities as it most often affects socioeconomically disadvantaged students (OECD, 2013b, 2013c, 2015). While it has decreased over the last decade, the percentage of students who reported having repeated a school grade is still high. In 2003, in OECD countries’ average, 20% of the assessed students reported having repeated a grade at least once, while in 2012 that figure dropped to 12% (OECD, 2013b).

The 2012 edition revealed that more than one out of three 15-year-old Brazilian students (36%) had repeated a grade at least once, thus reaching one of the highest repetition rates among countries participating in PISA (OECD, 2012). Between the 2003 and 2012 editions, the proportion of 15-year-old Brazilian students who had repeated a grade in primary education decreased, but the prevalence of grade repetition increased in secondary education, thus keeping the overall average stable.

In the 2006 edition of PISA, 40.9% of Brazilian students participating in the assessment were enrolled in the grade corresponding to their age group, i.e., in the 1st grade of secondary education. However, nearly the same percentage (40.6%) was still enrolled in the last two grades of primary education.⁵ Students who lagged behind their age group by more than two grades, i.e., retained in grades preceding 8th grade, were therefore not included in PISA.⁶

In 2015, normal-age students accounted for 37.7% of the 23,141 participants, and over-age students corresponded to 22.6%, a percentage that is still very high, but much lower than, or nearly half of that observed in 2006, including the 7th grade of primary education (Table 1).

With the expansion of primary education’s length to nine years, the PISA 2015 sample included students with eligible age from 7th grade of primary education onwards. The transition from the 8-year to the 9-year primary education spanned over three PISA cycles, but no significant differences are found in the distribution of students in these cycles, even with the inclusion of 7th grade in the 2015 sample (BRASIL, 2016).

Indeed, the percentage 15-year-olds enrolled in the grades assessed by PISA and the coverage rate, i.e., the number of students participating in the assessment divided by the estimated total of 15-year-old Brazilians, increased considerably in each PISA edition in Brazil (BRASIL, 2016). “While, in 2003, a total of 2,359,854 15-year-olds were enrolled from the 8th grade of primary education onwards, in 2015, over 2.8 million were in the grades eligible for the assessment”⁷ (BRASIL, 2016, p. 28, our translation).

The National Institute for Educational Studies and Research “Anísio Teixeira” (Inep) (BRAZIL, 2016) says that over the 15 years since the first administration of PISA in Brazil, the country has significantly improved the quality of official education statistics. “Until 2006, for example, the basic data collection unit for the School Census was the school”⁸ (BRASIL, 2016, p. 27, our translation). According to Inep (BRAZIL, 2016), by adopting the Educacenso,⁹ which establishes as research units, in addition to the school, the student and the teacher, the information about 15-year-old students eligible for PISA became more precise.

Also according to Table 1, there was a considerable increase in the percentage of Brazilian students enrolled in the final grades of secondary education: 18.5% in 2006 and 40.5% in 2015; it more than doubled. This difference, however, can be explained by the change in the month the PISA is administrated, which implies a change in the composition of students in relation to the various grades. The first edition of PISA (2000) in the country was administered in October, the following two (2003 and 2006) in August and, since 2009, Brazil has administered PISA always in May. Due to this change, more students in the 15-year-old age group were enrolled in more advanced grades. As ^{Klein (2011}) highlights, stability in the administration dates is important for better comparability of results and for a reliable diagnosis of the student population composition across the grades evaluated in PISA. According to the author, “it would be best to select students by the school age of 15 years in the country in question, and administer the test in a fixed number of months after the beginning of the school year. This rule should apply to all countries and all grades”¹⁰ (^{KLEIN, 2011}, p. 719, our translation).

Despite the “progress”, according to Inep (BRASIL, 2016), the figures also show that the educational work of including 15-year-olds into the school system is still a challenge for the country. “Based on the data from 2015, around 17% of them were out of school or enrolled in 6^th or in lower grades”¹¹ (BRASIL, 2016, p. 28, our translation). Indeed, from an international perspective (OECD, 2007, 2016), in terms of grade repetition, Brazil ranked second among all countries evaluated by PISA 2006 and third among the countries evaluated in the 2015 edition.

In general, the countries with the lowest average scores in international assessments are precisely the ones with highest age-grade distortion rates. The high distortion rates for countries like Brazil imply personal and social costs; an additional school year for the student represents a delay in completing basic education and entering the work world.

The following sections present the results of PISA 2006 and 2015 and the general average scores for OECD and Brazil in the four dimensions described in the methodological approach.

GENERAL AVERAGE SCORE IN SCIENCE

During most of the twentieth century, Brazil had poor educational indicators compared not only with European countries, but also with most Latin American countries (^{FRANCO et al., 2007}). The comparison of Brazilian students’ results places the country at a disadvantage compared to almost all countries participating in PISA. The results of performances in the Program in each knowledge domain are provided on a scale in which the average of OECD countries’ averages is standardized at 500 points, with 100 points of standard deviation. This means that approximately two thirds of the participant students scored between 400 and 600 points.

When comparing Brazil’s average (390) with that of the other countries participating in PISA 2006, the country’s overall performance in science is clearly not good. Brazil is among the countries with the lowest performance, ranking 52^nd among the 57 countries that took the examination in that edition, ahead only of Colombia (53^rd) when compared to our South American neighbors, which have socioeconomic conditions similar to ours.

Brazil’s average score in science has remained stable since 2006. There was an increase of approximately 10 points, i.e., from 390 to 401 points between 2006 and 2015, but this does not represent a statistically significant change (OECD, 2016). These results are similar to the historical evolution observed among OECD countries, where a slight decline in the average score (from 498 to 493 points in the same period) does not represent a statistically significant change either.

The Inep’s national report on data from PISA 2006 indicates, based on socioeconomic and cultural indicators, that it would not be reasonable to expect Brazilian students’ performance to be similar to the average of OECD students, though in science it should be about 30 points (30% of the standard deviation) higher to be within the expected for its average level. According to Inep (BRASIL, 2008), the positive association between students’ socioeconomic and cultural indicators shows that many educational systems have difficulties overcoming the determinants of students’ socioeconomic background. The main challenge faced by education systems is to ensure a good performance for the poorest students. Brazil is the Latin American participant with the lowest value: about 20 points below what is expected.

Based exclusively on the adjusted age-grade condition, i.e., considering only Brazilian normal-age students in the 2006 edition, the expected score for Brazil’s socioeconomic and cultural level was still not reached. Brazilian normal-age students’ average score is 408 points, therefore still 12 points short of what is expected. Such estimates are only met in the performance of students considered advanced. In other words, with an average 447 points, Brazilian students with at least one schooling year more than OECD students are able to reach an average score that, in the OECD’s view, corresponds to the necessary level for young people to fully exercise their citizenship (Table 2).

In 2015, the difference between Brazilian students in relation to 2006 remained practically stable. However, normal-age students had a lower average in science than that of the general sample, and also below the 2006 average. Advanced students, albeit in a greater proportion in this edition, also had a lower average in science than that of 2006. In sum, in the 2015 edition, as in the 2006 edition, only Brazilian students enrolled in the final grades of secondary education, i.e., with higher than expected schooling, were able to reach the initial levels considered by the OECD as essential for active participation in society (Table 2).

We could not close this section without looking at Brazilian over-age students’ performance. Having gone through the experience of grade repetition throws Brazilian students’ average to least half a standard deviation below the general average. Interested in the strength of the association between science proficiency and the situation of Brazilian over-age students, we implemented the following model through linear regression:

Where: β₀ is the average score of students who did not go through the experience of grade repetition; and β₁ indicates how much the science results of over-age students differ from those of non-over-age students.

Thus, Table 3 shows by how many points the average score of a Brazilian student assessed in PISA in 2006 and 2015 can be reduced when considering their grade repetition.

According to the data in Table 3, the regression model’s R² is 0.22 in 2006 and 0.15 in 2015. This means that the over-age variable explains about 22% and 15% of the average proficiency, respectively, in 2006 and 2015. Thus, this variable is better suited to the model executed for 2006, as its explanatory power is greater in 2006 than in 2015. It is worth noting that this is a simple linear regression model, without including variables that can control for the over-age effect on average proficiency. Anyhow, the data show that this variable explains the average proficiency better in 2006 than in 2015. Similar results were found by ^{Ikeda and Garcia (2014}), in which repetition alone explains the variance for Brazilian students’ average performance in reading in 21% (R2 = 0.21), in the 2009 edition.

Table 3 also shows that, in 2006, over-age students had around 87 points less than non-over-age students in the PISA science test, thus indicating that these students had, on average, 333.15 points. In 2015, the average for over-age students is about 3 points higher than these students’ average in 2006, and they have, on average, 81 points less than non-over-age students.

By comparing the 2006 averages with 2015, we can see that non-over-age students had a slightly lower average (about 417 points) in 2015 than in 2006 (about 420 points). In contrast, over-age students had a slightly higher average in 2015 (around 336 points) than in 2006 (around 333 points). Thus, the variation of averages between these two groups of students decreased in the analyzed period. The good news is that over-age students are managing to reduce, even if slightly, the difference in relation to non-over-age students. Future PISA editions may tell whether that decrease in inequality between groups of over-age and non-over-age students is a trend or a one-off event in the 2015 results. However, the bad news is that non-over-age students are failing to improve their average performance on PISA science tests.

The comparison between the individual performances of over-age and non-over-age students is a central issue mainly for the purpose of developing new policies (^{FERRÃO et al., 2017}). The results presented here place Brazilian students not only in a situation of extreme disadvantage compared to other students in the international context, but also below what would be the lowest level of performance established by PISA, as will be seen in the following subsection.

PISA PROFICIENCY LEVELS

To facilitate results interpretation, PISA established, in each domain or assessment area, several levels of performance based on the classification of the score associated with the skills that students must possess to achieve the corresponding score. According to the OECD itself, the classification has two purposes: cataloging students’ performance and describing what they are able to do.

The PISA 2015 science scale was divided into eight proficiency levels, six of which aligned with the levels defined in 2006. Each level’s description defines the necessary knowledge and skills to complete the tasks in the test and was designed based on the required cognitive demands. Students whose proficiency is below level 1 in 2006 and level 1a in 2015 are able to solve tasks at that level, but are unlikely to complete the scale’s upper levels. Level 6 includes the most challenging tasks in terms of knowledge and skills. Students with proficiency values ​​at this level are highly likely to perform tasks at this and other levels of the scale.

Table 4 shows the distribution of Brazilian and OECD students over the PISA proficiency levels. The PISA scale, as mentioned earlier, had six levels in 2006 - from levels 1 to 6 -, and eight in 2015 - from levels below 1b to 6. For the present study, we created level 0, which represents the group of students who did not reach proficiency level 1 established by PISA (an average score in science of at least 334.9 points) in both editions.

Table 4 shows that the highest percentages of Brazilian students in the general sample, which considers all students participating in the country, are at the scale’s lowest levels (27.9% at level 0 and 33.3% at level 1 in 2006, and 24.3% at level 0 and 32.4% at level 1 in 2015), which means a total of approximately 60% of Brazilian students at the Program’s most basic levels in both analyzed editions.

In large-scale assessments like PISA, it is predictable that few students will reach the highest levels. The common assumption is that most will be able to reach levels 2 or 3 in the proficiency scale. This can be seen in OECD member countries, where more than half of students (51.7% in 2006 and 52% in 2015) are at intermediate levels 2 and 3. Unfortunately, that is not the case, either for the Brazilian general sample, as we have seen, or for normal-age students in the country: students at levels 2 and 3 represented 42% in 2006 and only 35.6% in 2015. In other words, considerably less than half of normal-age students in Brazil were able to reach the level that the OECD defines as necessary for young people to be able to fully exercise their citizenship. Only Brazilian students enrolled in 2^nd and 3^rd grades of secondary education were able to reach the level established by the OECD (55.7% in 2006 and 54.3% in 2015).

The 2006 edition’s results place Brazil at level 1 and the OECD at level 3 in the PISA proficiency scale. According to the OECD (2007), at level 1, students have limited scientific knowledge, i.e., they can only apply it to a few familiar situations. They are able to present obvious scientific explanations and draw conclusions from explicitly presented evidence. At level 3, students are able to identify clearly defined scientific questions in a number of contexts: they can select facts and knowledge to explain phenomena and apply simple models and research strategies; interpret and use scientific concepts from different disciplines and apply them directly; and dissert about facts and make decisions based on scientific knowledge.

Brazil manages to rise by one level in PISA in 2006 and in 2015, but that requires at least one or two additional years of schooling. Our advanced students, i.e., enrolled in 2^nd and 3^rd grades of secondary education, reached level 2 in both editions of the Program and thus, according to the OECD, are able to draw on basic everyday and procedural knowledge to identify a suitable scientific explanation, interpret data and ascertain the topic addressed in a simple experimental project. They are able to use basic or everyday scientific knowledge to identify a valid conclusion from a simple set of data, and demonstrate basic epistemological knowledge by being able to recognize questions that can be scientifically investigated.

The situation of over-age students in Brazil, i.e., those enrolled in primary education, is very complex and converges with the findings of ^{Muri (2015}), which suggest that the age-grade distortion makes it impossible to achieve better results in science. Over 50% of Brazilian students who have experienced repetition are unable to reach the most basic level established by PISA. They represented 51.8% of Brazilian students at level 0 in 2006 and 51.6% in 2015. Over-age students are unable to advance in the proficiency scale. In both editions of the Program, they were retained at levels 0 and 1, and slightly over 10% managed to reach level 2.

Only a small proportion of students reach the highest proficiency levels in PISA - levels 5 and 6. At these two levels, students are able to identify, explain and apply scientific knowledge and knowledge about science to a variety of complex, real life situations. More recent data, i.e., from PISA 2015 (OECD, 2016), reveal that over half of all high-performing students in PISA live in only four countries: the United States, Japan, China and Germany. Next, we analyze the results of Brazilian students according to their competencies in science in PISA.

COMPETENCIES IN SCIENCE

Students from each country achieved higher or lower scores at certain competencies measured by PISA. In 2006, the Program assessed students’ ability in the following competencies: “identifying scientific questions”; “explaining phenomena scientifically”; and “using scientific evidence”.

The competency of “identifying scientific questions” involves recognizing and communicating questions that can be scientifically investigated and knowing what is involved in those investigations. It includes recognizing scientific questions such as what should be compared, what variables should be changed or controlled, what additional information is needed, or what actions should be taken so that relevant data can be collected. It also involves recognizing relevant characteristics of scientific research and identifying keywords to research scientific questions.

In the competency of “explaining phenomena scientifically”, students demonstrate their understanding by applying appropriate scientific knowledge to a given situation. This competency includes scientific description or explanation for phenomena and the prediction of changes, and it may also involve the recognition or identification of appropriate descriptions, explanations and predictions.

Finally, “using scientific evidence” means understanding scientific findings as evidence for demands or conclusions. It implies the ability to evaluate scientific information and reach conclusions based on scientific evidence and its future communication. In addition, it includes: the ability to select alternative conclusions in relation to evidence and communicate them; giving reasons for or against a given conclusion, based on provided data; identifying the assumptions made in reaching a conclusion; and reflecting on the social implications of science and technological development.

As mentioned earlier, the definition of scientific literacy in 2015 is based on the definition from 2006. In 2015, students participating in PISA were also assessed on three competencies, but the terminology in two of the three competencies changed compared to the 2006 edition. Only the competency of “explaining phenomena scientifically” remained unchanged. OECD technical reports do not contain any record of the change in the titles of the competencies; however, they make reference to other changes, such as the contents of the concepts of procedural and epistemic knowledge, which provide a more detailed specification of particular aspects that were incorporated or assumed in previous definitions (OECD, 2013c). The competencies assessed in 2015 were: explaining phenomena scientifically (recognizing, offering and evaluating explanations for natural and technological phenomena); evaluating and designing scientific enquiry (describing and appraising scientific investigations and proposing ways of addressing questions scientifically); and interpreting data and evidence scientifically (analyzing and evaluating data, claims and arguments, and drawing appropriate scientific conclusions).

Obviously, as with the general average score in science, Brazil and the OECD also had different performances in the different competencies assessed by the Program. Table 5 shows the country and OECD average scores for the different competencies assessed in 2006.

The competency in which Brazil had the best results in 2006, whether in the general sample or in the over-age, normal-age and advanced student groups, is that of “identifying scientific questions”. However, Brazilian results were below OECD countries’ average. “Identifying scientific questions” is the competency in which the OECD had the lowest average score. However, that average is up to 100 points higher than the Brazilian average, even when considering only the country’s normal-age students.

In the competency of “explaining phenomena scientifically”, Brazil’s average performance was identical to the country’s general average score in science (390). This is the competency in which, as a rule, students from OECD countries have their best performance on average. The lower results achieved by Brazilian students are related to the practical use of evidence provided by science. In this competency, Brazil had its worst performance, with 12 points below its general average. Even advanced Brazilian students lagged behind OECD students by more than 50 points, i.e., half a standard deviation.

The reality of Brazilian over-age students repeats throughout the competencies, as with the levels of proficiency established by PISA. The average scores were also lower than those of the general sample and those of the normal-age and advanced student groups. Having experienced repetition decreases a Brazilian student’s competency to “explain phenomena scientifically”, “identify scientific questions” and “use scientific evidence” by at least 52, 55 and 70 points, respectively.

In 2015, Brazilian students in general did better in the competency of “explaining phenomena scientifically” (Table 6), the same occurring to students comprising the over-age, normal-age and advanced groups. In other words, in all dimensions, Brazilian students performed slightly better in the competency of “explaining phenomena scientifically”. OECD average scores were the same for the three assessed competencies.

Again, even when comparing the Brazilian group of students with highest proficiency - i.e., advanced students - in their best competency - “explaining phenomena scientifically”, with 441 points -, our students are half a standard deviation below OECD students (an average 493 points), with a difference of 52 points.

In the other competencies, the differences reach nearly 60 points and, among over-age students, they reach over 150 points in relation to OECD students (one and a half standard deviation). Even the Brazilian group of normal-age students, with the same number of schooling years as OECD students, is at least one standard deviation below OECD students in any of the competencies assessed by PISA 2015. In addition to the competencies, we also analyze students’ performance in knowledge of the different science areas. This aspect is addressed in the next section.

KNOWLEDGE OF THE DIFFERENT SCIENCE AREAS

The content to be evaluated in PISA is selected from the major fields of physics, chemistry, biological sciences and earth and space sciences, according to three criteria: “usefulness of scientific knowledge in everyday life”; “relevance of science and educational policy over the next few years”; and “the need to combine knowledge with some scientific processes”.

The tasks in the PISA 2006 test involved scientific knowledge of two types: knowledge about science, which was divided into scientific research and scientific explanations; and knowledge of science, which focuses on knowledge of the natural world in content areas, such as: physical systems, living systems, Earth and space systems and technological systems.

As mentioned earlier, the main difference between Pisa 2006 and 2015 is that the notion of “knowledge about science” was explained based on its division into two components: “procedural knowledge” and “epistemic knowledge”. Thus, the three competencies required for scientific literacy in 2015 also require three forms of knowledge: “content knowledge”, which refers to the knowledge of facts, concepts, ideas and theories about the natural world as established by science - the equivalent of “knowledge of science” in the 2006 edition; “Procedural knowledge”, which is an understanding of the standard procedures that scientists use to obtain reliable and valid data; and “epistemic knowledge”, which defines the characteristics essential to the process of building scientific knowledge.

Some countries, like Brazil, achieved a substantially higher performance in “knowledge about science”, i.e., knowledge about the purposes and nature of scientific research and scientific explanations, than in “knowledge of science”, i.e., knowledge of the natural world and how it is linked with different scientific disciplines (Table 7).

While it was not able to reach OECD countries’ average score, the Brazilian general sample managed to exceed the 400-point barrier in “knowledge of science”, though specifically in knowledge of “living systems”, both in 2006 and 2015. In OECD average scores, results tend, albeit very subtly, to be better in questions requiring “knowledge of science” and, as with Brazil, the content with highest achievement is in the categories “living systems” (502), in 2006, and “Earth and space systems” (494), in 2015. According to the PISA report (OECD, 2008), such better performance in “knowledge of science” suggests that the curriculum has emphasized the transmission of specific scientific knowledge. However, if we compare the average scores in the subscales “knowledge of science” and “knowledge about science”, we can see, in Table 7, that Brazil, unlike the OECD, had a better overall performance in questions related to the latter type of assessed knowledge. This type of knowledge covered, in 2006, questions related to an understanding of nature, scientific work and scientific reflection and, in 2015, knowledge of the standard procedures that scientists use to obtain reliable and valid data, which defines the characteristics essential to the process of building scientific knowledge.

Unfortunately, in this subsection, we were able to present data for over-age, normal age and advanced Brazilian students only for the 2015 edition. The data for 2006 were collected from the website of the main PISA administration consortium - the Australian Council for Educational Research (ACER) -, since plausible scientific knowledge values ​​are not available in the database. The limitations regarding the aforementioned tool did not allow calculating average scores for the groups of students. Likewise, the average scores in technological systems, which also comprise “knowledge of science” in 2006, are not available either in the ACER database or in the PISA database.

As with the other categories analyzed here, knowledge of different science areas is also impacted by the student’s situation. In this case, in particular, both in “knowledge of science” and “knowledge about science”, over-age students were one and a half standard deviation below the OECD average score; normal-age students, a standard deviation; and advanced students, half a standard deviation (with two exceptions: the difference between the averages in living systems for normal-age and advanced students was 96 and 49 points, respectively).

CONCLUSIONS

The concept of literacy that underlies PISA is closely related to what is important in an individual’s everyday life, and its wider purpose is to assess young people’s ability to use their knowledge and skills to face the challenges of life in contemporary society. Thus, by adopting the term “scientific literacy”, rather than “science”, PISA underscores the importance that its science assessment places on the application of scientific knowledge to the context of real-life situations.

Interpreted from the perspective of assessing students’ ability to use scientific knowledge to identify questions of and make decisions about the natural and social world, this study’s results reveal a scenario of fragility and inequality in Brazilian students’ scientific literacy, which is aggravated by the recurring use of repetition in schools.

The comparison of science results in the PISA 2006 and 2015 editions places Brazil at a disadvantage in relation to almost all participating countries and not only the OECD. Brazil was ranked 52^nd in scientific competency among the 57 participants in PISA in 2006, and 63^rd among the approximately 70 countries in the Program in 2015.

The results show that, on the PISA performance scale, Brazil remains at level 1, while OECD countries on average remained at level 3. This means that most Brazilian students participating in PISA have limited scientific knowledge, i.e., they can only apply it to a few familiar situations. They are able to present obvious scientific explanations and draw conclusions from explicitly presented evidence. In turn, most students from OECD member countries participating in PISA 2006 are able to identify clearly defined scientific questions in a number of contexts. They can select facts and knowledge to explain phenomena and apply simple models and research strategies. They can interpret and use scientific concepts from different disciplines and apply them directly, as well as dissert about facts and make decisions based on scientific knowledge.

In addition, the article addresses the results of Brazilian over-age students in science, which allows shedding light on aspects of the educational process that Brazilian large-scale measures cannot reach.

The comparison of results achieved by Brazilian students in PISA made it clear that the age-grade distortion continues to be one of the most important dimensions of educational inequalities, although it is one of the main issues addressed by federal, state and municipal public policies in the country. We conducted our analyses based on the general OECD and Brazilian average scores in four dimensions: general, i.e., including all students participating in the test; over-age students (enrolled in primary education); normal-age students (enrolled in 1^st grade of secondary education); and advanced students (enrolled in 2^nd and 3^rd grades of secondary education).

Only Brazilian students enrolled in the final grades of secondary education and, therefore, with more schooling years than students from OECD member countries, were able to reach the initial levels considered by the organization as essential for active participation in society. Brazilian over-age students’ performance in science was below that of the general sample in Brazil. However, the difference between Brazilian over-age and OECD students reached a significant 150 points in some competencies, thus confirming findings of the specialist literature that show that grade repetition is not the best solution to teaching- -learning problems in Brazil, in addition to being a pedagogical measure that negatively affects performance in science in PISA as it deprives these students of the opportunity to learn what is typically assessed by the Program among 15-year-olds who should be attending the last grade of primary education or the first of secondary education.

In general, the countries with the lowest average scores in international assessments are precisely the ones with highest age-grade distortion rates. The high non-progression rates are expensive for countries like Brazil, since they imply significant educational and social costs resulting from longer schooling periods due to grade repetition, with the consequent postponement of completion of basic education and entry in the work world.

This study did not cover other aspects that are relevant to understanding education inequality, but it identified results of school practices associated with inequality in science learning that deserve attention from teachers, principals and education systems’ managers. Will the Common National Curriculum Base, a national policy focused on what students should learn, be able to align other policies on teacher education, production of teaching materials and assessments to the point of contributing both to improving teaching and to reducing learning inequalities in science? This is an urgent task in all areas of school knowledge.

This article was finished at a time that coincides with the pandemic caused by COVID-19, which materializes even more significantly the problems and challenges arising from the fragile scientific literacy of our students, in a context in which Brazilian scientific policy falls short of what is necessary, while the manipulation and politicization of science advance far beyond what is tolerable.