Fatigue effect and performance in standardized tests

Since the 1980s there has been an increase in the use of large-scale evaluations in Brazil (^{ALAVARSE, 2013}). Overall, these evaluations are of an exclusively additive nature, that is, they are used to select and certify students or for the accountability of educational systems (^{NEVO, 2011}). Examples of these evaluations in the Brazilian context are the Enem, Prova Brasil, the Saeb (National Basic Education Assessment System), and the Enade (National Assessment of Student Achievement – it evaluates the performance of students graduating from their bachelor’s degrees).

The number of participants in these evaluations excludes several methodologies that could be used in those processes and imposes other characteristics due to the difficult logistics in their implementation, including elaboration, administration, and correction (^{BAUER; ALAVARSE; OLIVEIRA, 2015}). Thus, it is common that large-scale evaluations are composed of close-ended questions, guided by a well-defined list of abilities or contents, and administered in highly-controlled contexts.

Even with these restrictions, these evaluations aim to verify students’ knowledge, that is, it is expected that each student’s result is a reflection of their cognitive abilities. However, even advocates of the methods employed in large-scale evaluations recognize that non-cognitive factors interfere in the measurement obtained by this type of instrument. For example, ^{Borghans, Meijers and Ter Weel (2008)} show that personality traits, such as intrinsic motivation and low aversion to risk, influence performance when variables such as time restriction and reward are modified.

^{David and Ferdous (2005)} tested if there was a difference in the performance of American students aged between 10 and 12 on questions as they appeared in different places in an evaluation. According to the authors, the hypothesis implicitly adopted by designers of this type of evaluation is the position of a question does not influence the test’s overall difficulty. However, when determining the difficulty of the questions using Item-Response Theory (IRT) on different workbooks of an evaluation (composed of the same questions, but in different orders), the authors found significant differences, in that the closer to the beginning, the easier the question is.

^{Meyers, Miller and Way (2008)} and ^{Albano (2013)} identified the same effect in other contexts. However, the authors went beyond measuring impact and proposed models that incorporate that effect to the performance calculation, following the IRT approach. That is, they proposed models that consider the position in which a question was presented to the participants as one of its parameters.

In the Pisa technical report conducted in 2000, ^{Adams and Wu (2002)} identified a difference in the difficulty level registered for a question when it is presented at different times in the evaluation. The difference is significant to the point that a correction factor was applied to obtain the final grade for each country. Although the authors did not explicitly relate that difference to the fatigue effect, the phenomenon seems to be related to it. More recently, ^{Borghans and Schils (2012)}, when analyzing data from subsequent editions of the Pisa, reported the same effect highlighted by ^{Davis and Ferdous (2005)}: students’ performance falls as the same question is presented later in an evaluation, regardless of the item’s difficulty level.

In a similar work, ^{Marchioni (2017)} investigated the fatigue effect in the Pisa and the results indicate that fatigue affects South American students more so than students from other countries. It is relevant to complement this observation with information already present in the Pisa technical report (2000). Comparatively, the authors conclude that the variation in a question’s difficulty is “very stable” (^{ADAMS; WU, 2002}, p. 157) when the countries participating are considered. Nevertheless, Brazil appears in first place in average number questions that the students did not even try to answer.

^{Sassaki et al. (2018)} examined data from the 2015 Pisa edition, focusing on the fatigue effect. The research design benefitted from detailed information thanks to the digital administration of the evaluation to a subset of students. The conclusion was that indeed, Brazilians are more affected by the fatigue effect than students from other countries, both from high performing and from countries with performance similar to our own.

The interference of this type of variable raises important questions, especially at a time when large-scale evaluations have been used at several levels and with purposes of great social relevance, such as the selection of students for university as is the case of the Enem, focus of this paper.

Research on the Enem

The Enem was created in 1998 with the goal of assessing the quality of learning at the secondary level. In 1999, it began being used as criterion for tertiary level entry, and in 2008 was reformulated substantially, being then founded on principles of Item-Response Theory, to serve as main access mechanism to several universities, including the majority of the country’s federal institutions (^{TRAVITZKI, 2017}). The Enem is conducted yearly, and the content of the test is divided over two days. Currently, the results can also be used to gain entry to some international universities, scholarships to private universities, and as a criterion for a secondary level graduation diploma⁸.

Since 1998, the data generated from the test, containing not only the responses, but also detailed participant information such as socioeconomic and geographical indicators, school trajectory, among others, are publicly available as microdata on the Inep’s website.

Analytically, the disaggregated nature of the information offers a unique opportunity to investigate large-scale phenomena related to the end-stage of Basic Education in Brazil. ^{Travitzki (2017)}, for example, used the microdata from the 2009 and 2011 editions to construct a reliability assessment goal for the tests, employing the same techniques that base the Enem’s elaboration. The author concluded that the 2009 Mathematics test presented insufficient reliability and several questions demonstrated unexpected empirical behavior in both editions.

Nonetheless, the effective use of microdata in applied research, whether that is for an assessment goal of the test or for an investigation of other phenomena, is still reduced. A search conducted in January 2020 in the Scielo portal using the terms “enem” and “microdata” returns only four papers, and none discuss similar topics to the ones we have proposed to explore in this work.

^{Lima et al. (2019)} present a systematic literature review of papers that use the data provided by the Inep regarding the Enem and the ENADE. With the aid of Google Scholar (https://scholar.google.com), having a wider scope than Scielo, ^{Lima et al. (2019)} reached a set of 54 works, published between 2005 and 2016, with 17 of these referring to the Enem. They were grouped into four categories according to the nature of their goals: content/knowledge, administrative, performance/achievement, and test/develop tools.

The papers in the first group investigated aspects related to the content in the Enem’s questions. The ones in the second group focused on aspects connected to management and access. Papers in the third group investigated the performance of specific groups or the relationship between performance with other variables. Lastly, papers in the fourth group emphasized the development of tools or methodologies that facilitate the interpretation and use of available information in the microdata.

Despite the papers in the third group having examined aspects related to students’ performance, the focus was to analyze and compare the performance of students according to some geographical or socioeconomic variable. Therefore, none of the articles analyzed by ^{Lima et al. (2019)} discuss similar themes to the one in this paper.

The paper presented by ^{Toffoli (2019)}, published after Lima et al.’s review (2019), comes closer to ours regarding the potential to contribute to the Enem’s improvement as a selection exam. According to the author,

Studies on large-scale evaluations are important to identify stages in the process that are not working as expected as well as to validate the steps with adequate results. In both cases, the aim of studies must be the improvement of the process at each edition of the exam. (^{TOFFOLI, 2019}, p. 4. Our translation).

^{Toffoli (2019)} analyzes the psychometric qualities of each question and of the test as a whole. Her conclusions are extremely concerning, given that the Enem’s results are more than just simple measurements and have very relevant social consequences (^{TOFFOLI et al., 2016}). Their results may be synthesized in the following statement:

In the 2015 Enem Mathematics test, the easiest item obtained the difficulty parameter at b = -1,98, and approximately 23,4% of participants have abilities lower than that level, indicating that these individuals could not answer any among the 45 items, only getting items correct by chance. (^{TOFFOLI, 2019}, p. 21. Our translation.).

In her study, ^{Toffoli (2019)} considered only the answers from one of the Enem’s workbooks, that is, it would be impossible to notice the fatigue effect when analyzing her data.

^{Travitzki (2017)} comes closer to our object of study when discussing his results. The author points out the presence of several questions in the 2009 and 2011 editions of the Enem that would have been considered inadequate according to criteria from Classical Test Theory and observes that a good portion of them were concentrated at the end of the test. The author offers two tentative explanations for this phenomenon:

One possible explanation is that previous items demanded too much work, leading candidates to greater mental exhaustion (or less available time) at the end of the test. Another possible explanation, not-exclusive, is that a good portion of the students may have not done the questions because they considered them too difficult, being thus more productive to invest the time for the test on other items. (^{TRAVITZKI, 2017}, p. 281. Our translation).

Note that the author does not discuss fatigue effect itself, but suggests that tiredness may affect participants’ performance in questions presented at the end of the test, to the point that they may be considered of dubious quality in psychometric terms.

The caveats expressed by ^{Toffoli (2019)} regarding the Enem, in conjunction with its great social relevance (^{TOFFOLI et al., 2016}), reveal the importance that the exam is place under scrutiny. Although it was not the main focus of his analysis, the explanations suggested by ^{Travitzki (2017)} already single out the relevance of considering the position of questions on the Enem when analyzing the test’s psychometric quality. In addition, the conclusion by ^{Sassaki et al. (2018)}, that Brazilian students are more susceptible to the variation of the position of questions than students from other countries, although observed in the new Pisa test, further reinforce the relevance of analyzing the fatigue effect on a test such as the Enem. Lastly, the dearth of studies identified in this review accentuate the unprecedented nature of this paper’s focus.

Question order in the Enem’s workbooks

The Enem is composed of four workbooks (visually differentiated by color) with the same questions but presented in different order. Figure 1 shows the first page of the Mathematics and its Technologies test for each of the four workbooks in the main application in 2016.

Source: Inep, 2016. Available at: https://www.gov.br/inep/pt-br/areas-de-atuacao/avaliacaoe-exames-educacionais/enem/provas-e-gabaritos. Accessed on: 14 April 2022.

Figure 1 Workbooks by color 

Operationally, the main objective of color diversification is to hinder copies or exchange of response cards since students sitting close to each other must receive different color workbooks. However, we did not find in the official documents justifications as to how the workbooks are put together. We also detected that the content of each page is fixed, but the order of the pages within a given knowledge field is altered from workbook to another. This means that some questions are always close to each other, regardless of workbook color, but a page may be placed in different positions. This is a similar configuration Pisa’s and was used by ^{Sassaki et al. (2018)} to identify the fatigue effect.

As discussed previously, this variation can affect the chances of correctly answering a question, which would indicate the interference of a non-cognitive factor in the performance of students. This is the phenomenon we will investigate, in an exploratory way, in this study.

Microdata from the 2016 Enem

The microdata format used by the Inep is analytically versatile, but it presents two relevant limitations regarding what we have proposed to investigate. The first, already mentioned, is the size of the files. The second is the format of the answers for each participant, given that they are not given as independent fields, but as a character sequence indicating which was the choice selected by the student, followed by a character chain with the respective answers for the questions. Technically, this format requires data pre-processing before it is actually possible to check any information regarding the answers given by the participants to the questions that made up the tests of each knowledge field contemplated by the Enem.

Thus, before importing the dataset to a statistical analysis software, we pre-processed it with a script in C language. The first step was to select only the students that were of interest to the research. Our selection included participants with regular attendance on both days, who were not taking the test to practice, with regular status in secondary education, and who worked on one of the workbooks used in the first application of the test¹¹. With these criteria, we ended up with a sample of 4.427.790 participants.

Afterwards, we altered the format in which the answers are stored in the microdata. The character chains with the answers for each question in the field of Mathematics and its Technologies were converted into 45 binary variables, one for each question, indicating whether the student got it right (1) or wrong (0). In addition, we separated participants per workbook (color), also including socioeconomic and demographic information. The decision to include these fields arose from the possibility to include them in subsequent analyses and, since our dataset will be publicly available, other researchers will be able to benefit from a wider set of information. After treatment, the data were analyzed using R, version 3.4.4¹².

Fatigue Effect

The exploration of the data started with identifying the position of each question (labeled by the numeric identification code adopted in the Inep’s microdata) in the four workbooks of the main application of the 2016 Enem. Then, the hit rate of each question was calculated separately per workbook.

Graph 1 shows the hit rate variation for the question with code 88786 in each workbook, identified by their colors.

Source: the authors, 2020.

Graph 1 Percentage of correct answers for question 88786 as a function of the position in which it was presented 

The value we are interested in is the percentage of correct answers per question. Theoretically, if there is no fatigue effect, the frequency of correct answers must be equal, regardless of the question’s position. However, as it can be observed, question 88786 appeared at the end of the Blue and Yellow workbooks with a mean hit rate of approximately 28%. When the same question was presented at the beginning of the test (Pink and Grey workbooks), the mean percentage of correct answers rises to about 34%. The difference in six percentage points equals a variation of 21,4% and it is sufficiently large to raise doubts about the validity and reliability of the test as an instrument for evaluating cognitive knowledge and abilities.

To estimate the degree in which this phenomenon affects other questions, we created an indicator especially designed to capture this trend, which we will call FE. A question’s FE is calculated from the difference between the hit rates when we maximize the distance of this question’s position. In the example presented in Graph 1, the largest distance is 39, which is the absolute difference between 44 (Blue workbook) and 5 (Pink workbook). Then, it is only a matter of calculating the absolute difference between 33,9% (hit rate in the Pink workbook) and 27,9% (hit rate in the Blue workbook) to find an FE of 6,0%. Table 1 shows the overall frequency of correct answers, the furthest distance, and the FE for all the questions. The information in Table 1 can also be visualized in Graph 2.

Source: the authors, 2020.

Graph 2 FE and distance for each question (n=45) in the Mathematics and its Technologies test for the 2016 Enem 

Each point represents one of the 45 questions; the X axis represents the furthest distance, the Y axis represents the FE, and the color represents that question’s hit rate (the lighter it is, the higher the hit rate). First, it is worth noting that the FE reaches 12% and this occurs in a question where the distance was 42 (point on the superior right corner of Graph 2). In addition, we have seven questions with an FE higher than 5%.

To explore how the two variables, furthest distance and FE relate, we used a linear regression on the dataset. The standardized coefficient of the line obtained is 0,326 (p-value=0,029; n=45), that is, if the distance of the same question increases in 10 positions, we should observe a mean variation in correct answers of 3,26%¹³. The coefficient of determination (r²), which is commonly interpreted as the total variance of the dependent variable explained by the set of explanatory variables, was 0,106, meaning that the distance between the items in the workbook is responsible for about 10% of the FE’s variation.

To ensure more robust results, we inverted the reasoning, to identify the FE from the shortest distance in which a question occurred among the different workbooks¹⁴. The expectation is that the FE calculated in this way will be very low, since we are comparing questions practically in the same position in different workbooks. Indeed, the minimum distances varied between 1 and 13 and we did not obtain an FE higher than 3%. The majority of questions (34) presented an FE below 1%.

Despite not being a homogenous effect, there are questions that have a considerably lower hit rate when presented at the end of the workbook than when they appear at the beginning. One possible explanation for that phenomenon, aligned with what ^{Davis and Ferdous (2005)} suggest, is that the participant tires throughout the exam and, when faced with a question that would require more cognitive effort, ends up with a worse performance than if that question were at the beginning of the test.

The fatigue effect we found with the aid of the FE indicator in the 2016 Enem, seems to us to be equivalent to the phenomenon identified by ^{Sassaki et al. (2018)} in the 2015 Pisa. The approach adopted in this paper, which innovated by incorporating the position of the questions in the different workbooks, complements the results found by ^{Travitzki (2017)} regarding the appropriateness of questions in the 2009 and 2011 editions of the Enem. The author identified, after analyzing only one workbook, that there was a higher concentration of questions with anomalous behavior at the end of the test and suggested as explanation for the phenomenon the tiredness of participants and the length of the test. The behavior of the FE that we have observed here offers new elements that may even help to understand the psychometric qualities of the Enem’s questions within the IRT’s approach.

Different groups suffer from fatigue on different questions

In Graph 2, it is also noted that questions with an overall low frequency hit rate have an overwhelming low FE, which suggests some interaction between these variables.

Of the 34 questions with up to 31% correct answers, 26 have a lower FE than 2% and the remaining 8 have an FE of, at most, 4%. That is, harder questions (with a lower hit rate) seem to cause a lower FE.

These observations, along with the fact that the Enem has a very heterogenous public, made us consider what the FE would be for participants with different performances.

To do so, we divided participants in groups based on the number of correct answers in the Mathematics and its Technologies test, not their final grade after using the IRT parameters employed by the Enem. Since the Enem’s questions are multiple choice with five alternatives, it is expected that a participant gets right, on average, nine questions, even if they answer them randomly. Thus, we divided our sample into five groups from the total number of correct answers, as presented on Table 2.

We investigated how the FE affects participants from different groups. For example, is it possible that a participant with a lot of correct answers (belonging to group 5), who likely has a higher level of knowledge and better preparation to take the test, is not affected as much by the fatigue effect? Similarly, is it possible that a participant who selects a lot of answers randomly (belonging to group 1), who likely has a lower level of knowledge and weaker preparation to take the test, is more affected by the fatigue effect?

After the groups were separated, we selected the five questions with higher FE for each, as seen on Table 3. This table does not contain 25 questions because some of them occurred for more than one group, such as 30865, which is among the five questions with higher FE for groups 1, 2, and 3. Highlighted in grey are the five questions that caused the highest fatigue effect for the group.

This table must be read by lines. For example, question 96833 was included because it had the highest FE for group 3 (17,94%). However, it caused a nearly null FE for groups 1 and 5, even though it appeared with a distance of 32.

The questions that had the highest FE for groups 2 and 4 are close in values but are not the same. The question with the highest FE for group 2 (32221) has a rate of 11,36% for that group and only 1,57% for the participants in group 4. On the other hand, question 85018 has the highest FE for group 4 (17,02%) and 0,01% for group 2. Comparatively, group 3 is the most affected, with a peak of 20,85% for question 18364. That was the second question in the Blue workbook and the penultimate question in the Pink workbook, with respective hit rates of 75,5% and 54,7%, among the participants of group 3. A very significant hit rate considering that it is the same question, just presented in a different order.

Considering the three middle groups, where 63,4% of participants are concentrated, with the exception of one question (45081), all present distance of 32 or more and the FE is high in the majority of cases. In turn, groups 1 and 5 show an overall lower FE.

All groups are affected, but different groups are affected on different questions. Despite there being some intersection among the questions in neighboring groups, its value varies considerably as the groups spread apart. Graph 3 illustrates the variation in the FE for the four questions that were chosen for having presented the highest FE in each group.

Source: the authors, 2020.

Graph 3 FE variation for four questions for the different performance groups 

Groups 1 and 2 had a maximum FE in the same question (32221) and it is observed that this value diminishes as the participants’ performance increased, being a question that practically does not affect the participants in group 5. Question 18364 is among the highest FEs for groups 1 and 2 and hits its peak in group 3. However, it is no longer among the five highest for group 4 and it has a very low FE for group 5. The peak for group 4’s FE was on question 85018, which did not hinder participants in groups 1, 2, and 5 too much. Lastly, the question that caused the highest FE in group 5, 60315, showed residual impact in all other groups.

Remembering the questions posited at the beginning of this section, we observed that students in groups 1 and 5 suffered less influence from the fatigue effect. Comparatively, group 3 was the most sensitive to this effect. It escapes our knowledge any papers that have analyzed data from the Enem considering different performance groups. The differences we have identified among the groups suggest that this approach is relevant to studies that seek to understand the performance of participants and/or assess the quality of the exam, such as the ones conducted by ^{Toffoli (2019)} and ^{Travitzki (2017)}.

FE and the questions’ difficulty

A possible explanation for the difference among FEs of different groups for the same question, observed in the previous section, is that questions that had a very low FE in a given group have a very high or very low frequency of correct answers in that group. That is, on one hand, participants get a question right, regardless of its position in the workbook, if it is easy for that group (high correct answer frequency). On the other hand, if the question is difficult for that group, the fact that it appears at the beginning or the end of the workbook does not alter the frequency of correct answers, given that, very likely, the participants in that group were not able to cognitively engage in solving the item. Consequently, we expect that questions with a high FE have an intermediary frequency of correct answers within that group.

To explore that relationship, we created Table 4, which presents the frequency of correct answers for the same questions as the ones in Table 3 (the highlights in grey indicate the questions that had the highest FE for the group).

Beginning with group 5, question 95265 stands out as the hardest question in the whole test, including the ones not presented in the table and, as verified by being in grey, is part of the set with highest FE. In addition to it, the other questions in grey for group 5 present an intermediary frequency of correct answers. The first 11 questions in the table had an 88% frequency of correct answers for this group and a low FE.

Three levels of difficulty can be seen in a more accentuated form in the questions in grey for groups 3 and 4: when the frequency of correct answers is too high, the question stops presenting a high FE value for that group and the same can be said when the frequency of correct answers is too low.

Note that for groups 1 and 2, the highest frequency of correct answers coincides (for the most part) with the questions with highest FE, since the hit rates for these groups never reach really high values (maximum of 45% for group 1 and 70% for group 2). On the other hand, the group of difficult questions (low hit rate) is quite large and presents an incredibly low FE.

A possible interpretation for this phenomenon is that the very difficult questions (low hit rate) cause little fatigue because participants do not invest a long time attempting to solve them or cannot cognitively engage with their resolution. On the other hand, questions with a high hit rate (too easy) also do not cause fatigue because participants manage to solve them even when they are at the end of the test, and they are tired. Questions at an intermediary hit rate level are those that are not too easy and can be solved quickly, nor too difficult, being beyond the candidate’s abilities, and thus highlight the effect of the fatigue caused by an evaluation as lengthy as the Enem.

That interpretation is aligned to what was suggested by ^{Travitzki (2017)} to explain the anomalous behavior for some of the Enem’s questions. Our analysis substantiates that suggestion by demonstrating that the fatigue effect differently affects groups with distinct performances according to each question’s difficulty.