QUANTILES, QUANTILE REGRESSION AND HIERARCHICAL STRUCTURE

The traditional methodology for analyzing factors associated with school achievement involves Regression Analysis, which focuses on determining how each factor affects the average achievement. Usually, the regression equation parameters are obtained through Ordinary Least Squares (OLS).¹

There is often an interest in determining whether a particular factor affects more strongly high- or poor- -achieving pupils. In this case, one way of conducting the analysis is using Quantile Regression (^{KOENKER, 2005}). In this procedure, analysis can be in terms of predefined quantiles (or percentiles), such as the median, first quartile and third quartiles of student performance distribution. If there is only one independent variable, then the quantile regression basically estimates several straight lines for the different achievement quantiles desired.

The best-known quantile is the median, q(50%), which separates the lower 50% of a sample from the higher 50%. In a distribution of performances assessed, the k-th quantile, here denoted q(k), may be understood as the value that delimits the poorest-achieving k% of pupils from the higher-achieving (100 - k)%. Figure 1 shows the position of the quantile 25%, denoted q(25%), which corresponds to the value on the scale that separates the poorest-achieving 25%

of pupils in a hypothetical distribution. The quantile q(25%) is also known as the first quartile or the lower quartile.

One advantage of using quantile regression for the median, rather than ordinary least squares regression for the mean, is that results will be more robust against outliers in response measures. ^{Chen et al. (2014}) point out that quantile regression is more useful than regression for the mean when the random variable has no normal distribution. This type of regression can be seen as a natural analogy to the use of different central tendency, dispersion and asymmetry measures to obtain a more comprehensive and robust analysis (^{KOENKER, 2005}). It is also noteworthy that quantile regression allows analyzing any quantile.

Figure 2 shows various pupil performance quantiles for the comparison between two groups, i.e., pupils with often present teachers and with often absent teachers, according to the answers in the questionnaire administered to the pupils. This figure shows the quantiles through box plots - which is a common representation of medians, quartiles and extremes - and through quantile regressions with the independent variable indicating the attribute in question (0 = often present teachers, 1 = often absent teachers).

Figure 2 shows that quantile regressions allow evaluating not only the possible differences in the central position of the distribution, but also whether the effect of the factor of interest is higher on poor-achieving or on high-achieving pupils. The example above shows that the performance expected (median) of pupils whose teacher is often absent is lower than that of pupils whose teacher is often present. However, it seems that the effect of an often absent teacher is approximately the same for the quantiles analyzed (roughly parallel lines). Situations where the lines at the lower quantiles are steeper (have greater slopes) than at upper quantiles indicate that the factor has a greater effect on poor- -achieving students. ^{Martins and Pereira (2004}), ^{Davino and Vistocco (2008}), ^{Rangvid (2008}) and ^{Almeida (2014}) present applications of quantile regression in education.

Another common characteristic in educational data analyses is their data hierarchical structure, i.e., pupils are subunits of schools, which may be subunits of boards, districts or other groups. For this reason, we can have groups of students within schools, possibly generating correlated measures within each group. We thus have a source of intraschool variation and a source of variation between schools. The analysis can be affected by intra-group correlations, which makes hierarchical models a good way of portraying this data structure.

Finally, to determine whether a factor has a greater effect on high-achieving or on poor-achieving students, this work focuses on quantiles of performance distribution, considering the hierarchical structure of the data. Hierarchical multiple regression modeling allows isolating the effect of each factor and separating the variation between schools from the variation between students at the same school.

In traditional models, factor effects are evaluated in terms of differences in the expected values. The present study innovates by examining how effects can differ in terms of quantiles, thus allowing to assess effects at different academic achievement levels.

METHODOLOGY

In theoretical terms, the k-th quantile, q(k), is given by the equation P{Y ≤ q(k)} = k, where Y is a continuous random variable. ^{Koenker (2005}) and Koenker and Bassett (1978) present a way of finding the median - i.e., q(50%) - using the least squares technique. Based on a set of values (y ₁, y ₂, ..., y _n ), the value of θ which minimizes the sum is obtained:

The relationship extends to the other quantiles - i.e., q(k) -, considering different weights for the positive and negative residues (r _i = y _i −θ ):

The quantile q(k) is the value of θ that minimizes relationship (2). This method of minimum absolute errors is equivalent to that of maximum likelihood when errors are assumed to have a Laplace distribution, as described by ^{Geraci and Bottai (2014}). According to ^{Yu and Zhang (2006}), the Laplace distribution is defined by three parameters: one of location, one of dispersion and one of asymmetry; this allows conducting the analysis when the distribution of the variable studied is symmetrical or asymmetric.

Based on ^{Geraci and Bottai (2014}), we used in this study the hierarchical quantile regression model described below:

where:

Y_ij is the dependent variable (achievement) for pupil i at school j;

β _oj ^(k) is the k-th quantile of school j, conditioned to the

other variables of the model that are equal to zero;

β _I ^(k) is the effect of l-th variable for pupil at the k-th quantile (l = 1, 2, ..., L);

k is the schools’ mean k-th quantile, conditioned to the other variables of the model that are equal to zero;

y _o ^(k) m γ is the effect of the m-th variable for school at the k-th quantile (m = 1, 2, ..., M);

e _ij ^(k) is the random error associated with i-th student of school j, supposedly with a Laplace distribution with location parameter equal to zero, dispersion equal to σ and asymmetry equal to k;

u _j ^(k) is the random term associated with school j, supposedly with the normal distribution mean of zero and variance σ_u ².

We controlled for the heterogeneity of schools through the random effect at β _oj ^(k) i.e., each school can have a different k-th quantile. Generally, there could be a random effect at some of the effects of student variables, i.e., at β _l ^(k) (l = 1, 2, ..., L). For example, school failure may be assumed to be different depending on the school, as some schools may have treat differently the pupils who were retained.

The model parameters can be estimated through the maximum likelihood method. To that end, we used the package lqmm (GERACI, 2016) do software livre R (^{R FOUNDATION FOR STATISTICAL COMPUTING, 2017}).²

There is not an extensive literature on the subject, but there are other proposals to introduce a hierarchical structure into quantile regression, such as in ^{Chen et al. (2014}) and ^{Hassan (2014}).

It is worth mentioning the approach of ^{Galarza, Lachos and Bandyopadhyay (2017}), who propose improvements in the algorithm to obtain maximum likelihood estimates in hierarchical quantile regression, offering a new package in at R, i.e., qrLMM, which we have studied.

DATA AND VARIABLES

We applied the methodology proposed here to the Pará State Education Assessment System (SISPAE) which was launched in 2013 and evaluates pupils in Portuguese and Mathematics. The 2016 edition assessed pupils in the 4th and 8th grades of primary education, as well as pupils in the 3rd grades of secondary education. The assessment involved students, teachers and principals of 3,534 schools in the 144 municipalities in the state of Pará. Of those, 814 are state schools, 2,714 are municipal schools and six are the so-called Rural Family Houses. Of the initial estimate of 563,413 pupils registered for the assessment, 376,830 actually took the test in Portuguese and 376,684 in Mathematics.

The 2016 edition of SISPAE included the collection of data on pupils’ characteristics for all grades assessed, as well as on teachers, pedagogical advisors, education experts, principals and schools, in order to outline the profile of respondents and obtain data for a more detailed analysis of factors associated with pupil achievement. To that end, questionnaires were administered to: 270,424 students; 15,250 teachers; 3,094 pedagogical advisors; 19,769 education experts; 3,509 principals; and 3,509 schools.

With regard to associated factors in the SISPAE (^{PARA, 2017}), we describe various measures and indicators based on context questionnaires, particularly those answered by the students. We selected these measures and indicators based on the literature on associated factors of school achievement, such as the studies of ^{Alves and Soares (2008}, ²⁰¹³) and ^{Laros, Marciano and Andrade (2010}), as well as technical reports on large-scale assessments carried out in other Brazilian states. In this paper, we selected the indicators that were statistically significant, i.e., the ones most strongly associated with performance levels, according to the analysis of SISPAE (^{PARA, 2017}). Since our focus was to show a new analysis approach, we decided not to use a large number of factors.

We constructed various measures using item response theory (IRT) with degree scale models (^{ANDRADE; VALLE, 2000}; ^{AYALA, 2009}), three of which were used in this paper: SISPAE 2016 (^{PARA, 2017})

Measurements were also aggregated, in terms of the means, for the classes of the pupil’s school year. In this paper, we standardized such measures with a mean of zero and a standard deviation of 1 in relation to the group of pupils studied. We standardized the aggregate measure in relation to the group of schools studied.

We also considered the following variables indicating the presence of factors associated with academic achievement: student sex; grade retention indicator; indicator of homework done, among others. Table 1 shows the full list of factors analyzed in this study.

RESULTS AND DISCUSSION

In this section we present and discuss the results of hierarchical quantile regressions it was conducted with the Portuguese and Mathematics tests for the 8th grade of primary education in the state of Para in 2016.

The estimation was based on a set of 1,808 schools and more than 66,000 pupils, which correspond to the observation units with answers in the variables studied.

To better illustrate the information that can be obtained through our approach, initially we will present an analysis performed through common hierarchical regression, based on the mean value of performance levels. Table 1 shows the expected effect of each factor included in the analysis. The estimation method was maximum likelihood, using lme4 (^{BATES et al., 2015}).

All estimates presented in Table 1 are statistically significant (p < 0,05).³ By analyzing effects estimates, we found that school climate and school relationship levels (SCL and SRL, respectively) have a positive effect on academic achievement in the population studied. In figures, for each additional standard deviation of pupil-perceived school climate, the mean performance in Portuguese tends to be two units higher.

In general, girls did better in Portuguese, and boys, in Mathematics: the estimated effect on girls’ means compared to boys is of 10.6 in Portuguese and -5.0 in Mathematics.

The mean performance levels for the group of pupils who had been retained were significantly below the levels for the others: estimates were -12.8 in Portuguese and -10.5 in Mathematics, i.e., pupil retention due to poor performance (or other reasons) does not tend to bring these pupils to the same level as the other pupils in the following years.

The results above are consistent with other studies on associated factors, such those of ^{Laros, Marciano and Andrade (2010}) and ^{Alves and Soares (2008}). However, these effects are observed at the mean level of performance, leaving unanswered questions such as the following: Can the effect of school climate be greater on poor-achieving pupils? Does the difference between females and males occur to a greater degree on poor- than on high-achieving pupils, or vice versa? And is the effect of a homework-concerned teacher greater on poor- than on high-achieving pupils or vice versa? Overall, what we try to answer in this paper is whether a given effect is greater on high- or on poor-achieving pupils. To that end, we conducted the analysis below based on the quantiles of the distribution of performance levels.

Table 2 shows various quantiles of SISPAE 2016 scores in the SAEB scale. So, for example, in the Portuguese test, the quantile 10% equals 147.4, which means that the performance of 10% of pupils in Portuguese is equal to or below 147.4 points. With regard to the quantile 90%, Table 2 shows that the performance of 90% of pupils in Portuguese is equal to or below 256.1 points, or put differently, 10% of students perform above 256.1 points. Therefore, q(10%) separates the poorest-achieving 10%, while q(90%) separates the highest-achieving 10%.

In this section, the discussion focuses on how much the different values of the explanatory variables - or the presence of factors that usually affect achievement - can change the values of quantiles and, moreover, whether the effect is greater (or smaller) on the quantiles that separate poor-achieving pupils, or on the quantiles that separate high- -achieving pupils.

Table 3 presents the estimates for the effects of the studied factors on some quantiles of the distribution of performance levels in Portuguese. Overall, we only present maximum likelihood estimates that had statistical significance at the 0.05 level. Tables with standard errors and tests of significance for these quantiles are shown in the Appendix (tables A1 to A5).

The effect of family socioeconomic status (SES) on pupil achievement is very well-known. Table 3 shows that an increase in pupil family socioeconomic status (SES) standard deviation drives an increase of 3.7 points in the median performance, an estimate similar to that obtained with the model for the mean (Table 1), which is reasonable, since both measures are of central tendency for the distribution of performance levels. However, with quantile regressions, the effect of SES is greater on higher-achieving pupils: in the quantile 10% (the point that separates the poorest-achieving 10% of pupils), the estimated effect of socioeconomic status (SES) standard deviation was 2.5 additional points in the test, whereas in the quantile 90% (which separates the highest- -achieving 10% of pupils), the estimated effect was of 4.6 additional points. By analyzing the values for grade/school SES, we can see that the relationships are similar to what occurs with pupil family SES. Table 4 shows the quantiles for the distribution of performance levels in Mathematics. The relationships are very similar to Portuguese for most of the variables/factors analyzed.

Tables A1 to A5 show the estimated values of the coefficients of the quantile hierarchical regressions, including the respective standard errors and the p-values for the statistical test associated to the null hypothesis in which the coefficient is null, i.e., the variable or factor under analysis has no significant effect.

It is worth noting that some coefficients had no statistical significance (p-value greater than 0.05) and, therefore, were not explained in tables 3 and 4 above; however, tables A1 to A5 list all the results obtained through the adjustment of the models.

Figures 3 to 7 reproduce the effect estimates in each quantile for the factors in which important differences were observed, Portuguese and Mathematics, including 95%

confidence intervals.

Figure 3 shows the effects of pupil family socioeconomic status. In both areas, the intervals at the quantiles 10% and 90% are disjoint, indicating a significant difference; the same occurs between the lower quartile - q(25%) - and the upper quartile - q(75%). In other words, the effect of SES is significantly greater on higher-achieving students.

One well-studied factor in large-scale assessments is grade retention. In both tests, we found a negative effect of “pupil has been retained”, with a reduction of more than 10 points in the median compared to the group that had not been retained. It is also noteworthy that this effect is greater on higher-achieving pupils. In the quantile 90%, which separates the highest-achieving 10% of pupils, the difference between the two groups was 15.1 points for Portuguese and 14.3 points for Mathematics (Figure 4). It is worth noting that there are far less cases of retained high-achieving pupils, but it is also worth considering the influence of some other uncontrolled factor, such as student/family in previous years, which led to retention.

Retained students were found to perceive negative effects associated with “teacher is often absent” and “teacher gives more attention to good pupils” (tables 3 and 4). There is no clear tendency of increase or decrease in these effects towards larger quantiles, i.e., the effects of these indicators are negative for high- and poor-achieving pupils alike.

The positive effect of “teacher is concerned with homework” is slightly higher for higher-achieving students. This suggests that higher-achieving students are making better use of homework and correction by the teacher (Figure 5).

An interesting point is the strong negative effect when pupils perceive that “Learning depends more on teacher than on pupil”. By using quantile regressions, we found that this effect is greater on higher-achieving pupils. In Portuguese, for students in the quantile 90%, the difference is around 12 points. In contrast, for those in the quantile 10%, the negative effect is around 6 points. Perhaps this behavior is characteristic of poor-achieving pupils, whose group homogeneity (whether agreeing or disagreeing with the statement) leads to a smaller difference of performance, the opposite occurring with the higher-achieving pupils (Figure 6).

An interesting point which can be seen in various studies on large-scale assessment, is that in Mathematics boys have a better performance, while in Portuguese it is girls who perform better. Figure 7 shows the effects of the sex factor in different quantiles, both for Portuguese and Mathematics.

The novelty about our finding is that in Mathematics the difference between sexes is higher among the higher- -achieving pupils, with the boys performing significantly better than girls (almost 10 points in the quantile 90%). On the other hand, in Portuguese, where girls stand out, the difference is greater among poor-achieving pupils (more than 10 points at the lower quantiles). Considering Portuguese and Mathematics together, our analysis suggests that boys are in a more extreme position than girls, since in Portuguese, where their performance is usually worse, the difference is greater among the poor-achieving; in Mathematics, where their performance is usually better, the difference increases for the high-achieving.

CONCLUSION

There are few studies on factors related to academic achievement, which use quantile regression, especially hierarchical quantile regression. In a way, this is understandable, since hierarchical quantile regression is not yet a well-known, ‘mainstream’ method. Another reason why few studies use it is that it is not implemented in main commercial software in the field of statistics. However, our analyzes presented good evidence of factors that have a more significant influence when comparing high- and poor-achieving pupils, thus providing school management with valuable insights.

In this study, we have shown, for example, that pupil socioeconomic level causes greater differences among high- -achieving pupils; the same holds for the positive effect of the “teacher is concern with homework” factor. Poor-achieving pupils seem to be more dependent on the teacher, since agreement with the “learning depends more on teacher than on pupil” statement has a high negative effect on the various quantiles of the distribution. Using the quantile regression approach, we found that this negative effect is stronger on higher-achieving groups, i.e., high-achieving pupils who agree with this statement tend to perform significantly below high-achieving ones who disagree.

Our results should encourage further studies with other populations, which might confirm the several relationships discussed in this paper. Moreover, there are public policies designed to mitigate academic achievement differences among pupils through actions to improve performance, especially for those with greater learning difficulties. Hierarchical quantile regression can be a good tool to objectively evaluate the effect of these policies because it measures the effect of factors of interest on different academic achievement levels, not only on their expected value.