INTRODUCTION

The Progress Test (PT) is a type of serial, longitudinal assessment of students’ knowledge, in which students from medical schools periodically take, on the same day, a test aimed at assessing the knowledge expected at the end of undergraduate school. Among the several uses of the PT in the teaching and learning process, the results of these tests are used for students to verify their individual performance during undergraduate school, and for each school to evaluate the performance of the students, comparing them with students from the other participating schools^1)-(3. The proportion of students absent from the PT varies greatly between different schools. It is not known whether this absence occur randomly or due to some systematic characteristic of the absentees, which could affect the global representativeness of the PT results.

The lack of information about the characteristics of the missing data in an assessment, such as the Progress Test, can generate biased information^4)-(7. If the students’ absence occurs at random, the results of this evaluation may be representative of the school; but if the absence occurs systematically, more predominantly, for instance, among students with high or low performance in the medical course, then it is likely that the PT result is not truly representative of the school⁶.

Either independently, or with the support of the Brazilian Association of Medical Education (ABEM, Associação Brasileira de Educação Médica), groups or regional consortia of medical schools prepare and apply a Progress Test per year⁸. In Brazil, student participation in the PT is not mandatory in all schools^9),(10. In other countries, this assessment is applied two to four times a year. In the Netherlands, for instance, the test is administered four times a year and participation is mandatory; the results are included in each student’s individual curriculum and the PTs can be a substitute for the professional licensing exam^2),(11.

Although it may have an influence on school results, the causes of PT absenteeism in Brazil, the characteristics of absent students and the impact of the proportion of absentees on the PT results are not studied. Little value is assigned to the existence, or not, of any bias in the results, due to some systematic characteristic(s) of the absent students. There is also little concern about how to deal with missing data, even though there are statistical techniques, with robust literature, that can be used to impute presupposed values to the group of students that were absent from the PT^12)-(14.

Some techniques for imputing data to absentees, using the average of the ones present at the test, are not very concise⁴, but others, using multiple regression, are more precise^4),(14)-(16. These techniques are widely used in planning, research and evaluation of educational performance^{4),(6),(14),(15),(17),(18}. Students’ grades and academic performance indicators can be used to verify whether there is any systematic difference between being present and being absent from the PT. Such indicators can also be used to estimate and impute probable grades in the PT to absent students, aiming to make the results of this type of assessment more representative of the school.

With the objective that periodic evaluations, such as the PT, can, in the future: i) contribute to improving public policies to control and improve the quality of medical education and ii) influence or determine the practice of medicine by trained physicians, it is important that their results in the PT are truly representative of the training of all medical school students. Thus, the issue of absent students must be very well valued to consolidate trust (representativeness) in the results of the PTs and consolidate the legitimacy of the impact of this type of evaluation in medical education and future practice of medicine.

METHODS

RESULTS

In 2019, of the 565 UFSC medical students who could participate in the PT, 443 (78.4%) attended the test and 122 (21.6%) did not attend it. Regarding gender, of the 306 male students, 79 (25.8%) were not present and of the 259 female students, 43 (16.6%) were absent. It is verified that the proportion of men who did not attend the 2019 PT is statistically higher than the proportion of women (p <0.01).

The averages of the academic performance indexes (EI, AAI and PAI) in the medical course, in the semester when the PT was applied, of students of both genders, present at and absent from the Progress Test, are shown in Table 1.

The three academic performance indexes used at UFSC indicate that students who attended the PT test in 2019 had better performance in the course when compared to those who did not attend it. These differences were statistically significant for all three indicators.

The same comparison, using only the “EI” index, was carried out in the different phases of the medical course - Table 2. The comparisons in the phases, using AAI and PAI, showed similar results and will not be presented here.

In ten of the eleven phases (or semesters), the averages of the EI academic performance indicator were higher among the groups of students who attended the PT, when compared to the groups of absentees; differences were significant at four phases.

Using the three indicators of academic performance of the absent students, in the phase that was being attended, the imputation of the probable grades in the PT to the absent students was made by linear regression. The average scores at the PT of the present students, the percentages and average scores attributed to the absent students and the differences between the averages of those present and the ones attributed in each phase of the medical course are shown in Table 3.

In nine of the eleven analyzed phases, the averages of the PT grades of the students present at the test were higher than the averages attributed to the absent students. In this analysis, the significant differences increased to six phases, with a predominance in the final phases of the course. In phases five and twelve, with a relatively low proportion of absentees, the scores attributed to those absent from the PT were higher than the scores of the PT participants, but the difference is not statistically significant.

With the data, percentage of absentees and difference in the students’ grades in the PT and the grades attributed to the absentees described in Table 3, it is possible to verify whether there is a relationship between the two variables (Figure 1).

Source: Prepared by the authors.

Figure 1 Relationship, in the phases of the course, between the differences in the scores of students present at and absent from the Progress Test with the percentage of students absent from the 2019 PT 

The regression showed that there is a positive linear relationship between the differences in the scores of students present at and absent from the Progress Test with the percentage of students absent from the PT. The “R” correlation was 0.62 (p < 0.0001) and the coefficient of determination “R²” was 0.38 (p < 0.0001). That is, there is a tendency that, as the percentage of absentees increases, the greater the difference between the scores of those present at the PT and the scores attributed to the absentees, as seen in Figure 1.

DISCUSSION

In the analyzed PT, 21.6% of UFSC students were absent from the test. This percentage was higher than those found in other studies on PT in Brazil, which describe less than 10% of absentees^25)-(27. In the international literature, a Dutch article, which analyzed the use of PT in post-graduation for 14 years, reports an increase throughout the studied period from 1% to 16% of radiology residents requesting exemption from participation²⁸. Regarding the significantly higher proportion of male students missing the 2019 PT, we did not find data in the national or international literature to compare our results. Differences between genders are observed in the literature on medical education, with men having a higher rate of absenteeism in classes, which is associated with higher failure rates^29)-(31 and lower educational performance^29),(32),(33. Women have a lower rate of conflicts inside medical schools³².

The analysis of the scores of three academic performance indicators (EI, AAI and PAI), comparing students present at with those absent from the PT, show that the group of absentees has worse academic performance. Using one of these indicators, called the Enrollment Index (EI), to separately analyze each of the 11 phases of the course, we observed that in 10 phases the group that did not attend the PT had worse academic performance than the group that participated in it. This measurement strongly suggests that absence from the PT does not occur randomly. The group of students who were absent from the PT showed a systematic tendency to be consisted of students with lower academic performance. In the literature on the PT, we did not find any other study that compared characteristics of students present at and absent from the test.

The finding that PT absenteeism does not occur randomly leads to two reflections. At the individual level, the literature on education suggests that student characteristics such as absenteeism from classes, social isolation, depression and language difficulties are risk factors associated with poor academic performance³². The present study suggests that absence from the PT could also be a sign of academic difficulty; therefore, it could, or perhaps, even should be considered a warning for early interventions to support this group of students, potentially with difficulties, aiming at pedagogical and social inclusion interventions.

At the institutional level, if the absence from the PT occurs in a systematic way, consisting mostly of a group with lower academic performance, then, with the increase in the proportion of absentees, the average performance in the PT of those present at the test tends to “apparently” increase the school result, making the latter biased^{6),(13),(15),(34}, as shown in this study. There is no certainty, but researchers working with missing data statistics suggest in systematic absence that, when the proportion of missing students is greater than 5 to 10% of students in a course, the results will be biased (in this study it was 21.6%); when, however, the absences occur randomly, they do not significantly affect the results^6),(20.

Therefore, it is reasonable to suggest that institutions with 10% or more of students absent from the PT establish policies to increase student participation. Making participation in the PT mandatory would be the first step. Scoring participation in the PT, in the school records of each student, would be another idea that could be discussed. To minimize the bias caused by missing data in school results, the use of some statistical strategy, such as data imputation, could increase the actual “representativeness” of the PT results.

The occurrence of missing data, shown in clinical or educational research, reduces the sample size and variability, decreasing the statistical power and external validity of the studies^15),(24. On the contrary, the use of the multiple data imputation technique to calculate the absent students’ probable grade in the PT and the inclusion of these results preserves the size and variability of the samples. Reviews on the use of imputation of missing data in educational research can be found in the national^4),(15 and international^6),(17),(18 literature.

Overall, the missing data are classified as: Missing Completely at Random - MCAR, which do not change results; Missing at Random - MAR, which may or may not affect the results (depending on whether random absences affect different extracts under study in different manners); Missing Not at Random - MNAR, which affect the results¹², such as those found in our study. However, after imputation of data, we observed that, specifically in the fifth and twelfth phases, the absentees’ performance was better (but not significantly) than that of students present at the PT. Therefore, it cannot be said that the absentees are always the students with the lowest academic performance.

The linear regression between differences in grades in the PT and the percentage of absentees, as seen in Figure 1, show that there is a relationship between the percentage of absentees and the average performance in the PT. The more absentees, the higher the average grade of those present at the PT. Therefore, it is reasonable to consider that, when internally comparing the results of a school²⁵, or comparing the results between different schools or consortia⁸, information on the percentage of absentees should be included; moreover, that the missing data should be imputed, so that the expression of the results is closer to the representativeness of the global performance of the assessed schools.

Despite showing that absence from the PT does not occur at random, but systematically, that there is a relationship between the percentage of absentees and the difference between the grades of those present and the grades attributed to the absentees, and that it affects the results, the present study does not demonstrate as of what percentage of absentees the data do not represent the real values of the school. To more accurately verify which percentage affects the data, much larger samples would be needed than those used in the present study.

Another observed limitation is that it is a single-center study, based on only one test. In the period from 2011 to 2018, the absenteeism rates at the PT at UFSC ranged from 8.0 to 37.7% (average = 18.2%; SD = 10.5). The percentage of absentees of 21.6% found in this research is close to the average of nine years of PT at UFSC. Traditionally, the highest proportions of absentees were found among the students attending the seventh and eighth semesters. In 2019, there was an unusual absence of 47.9% of students in the ninth phase. The attempt to understand why this happened, and who the absentees were in the ninth phase, originated the present study. A similar analysis in relation to the absentees should be carried out in other schools, to confirm the external validity of the findings of this study.

Currently, in Brazil, there are groups willing to support the implementation of licensing exams for the exercise of the profession^35),(36. This discussion is a heated one and occurs in many countries around the world, even without any research unequivocally showing the advantage of this type of assessment^37),(38. As the licensing exam takes place at the end of undergraduate school, our fear is that it will (belatedly) fall only on the student and will not have consequences for the schools and on the quality of teaching. With or without licensing, it seems obvious to us that serial, longitudinal assessments, such as the Progress Test or the National Serial Assessment of Medical Students - ANASEM (Avaliação Nacional Seriada de Estudantes de Medicina)³⁹, discarded by the federal government, should be encouraged and resumed. Assessments from the beginning of undergraduate school would allow students to implement an early correction of their paths and allow schools to do the same. They could greatly contribute to the quality of medical education.

FINAL CONSIDERATIONS

The current study shows that there is a positive correlation between male gender and absence from the PT (male students were more often absent); that the absences did not occur at random, but systematically, with the group of absentees predominantly consisting of students with lower academic performance. This suggests that the absence from the PT can be used as a potential marker and warning for students with poor performance, to establish policies to encourage and include them, from the initial or intermediate phases of the courses. It also shows that there was a correlation between the percentage of absentees and a possible change in the “actual” result (global representativeness) of the PT. The technique of multiple imputation of data to absentees can be used to estimate what the school grades would be if all students participated in the test.

It remains clear the importance of encouraging strategies to reduce the percentage of PT absentees in Brazil, of taking better care of this group of students and promoting actions to increase the representativeness of the PT results. This would contribute to making them more useful and reliable, valuing longitudinal assessments of students and courses and, consequently, contributing to public policies that will improve medical education in our country.