INTRODUCTION

Progress tests are comprehensive longitudinal assessments used to measure students’ knowledge throughout a curriculum that allow, when properly designed and applied, observation of the individual growth pattern over repeated tests and comparisons of cohorts^1)-(6.

In Brazil, at Escola Paulista de Medicina - Universidade Federal de São Paulo, SP, the progress test was introduced in 1996^7),(8. This assessment has been carried out annually for students from the 1^st to the 6^th year of medical school at several institutions, either alone or in partnership^1),(9. Additionally, numerous studies have been carried out aiming to verify the students’ performance on the progress test^7),(10),(11.

Furthermore, an important question that arises is about the validity of taking the progress test from the 1^st to the 6^th year, taking into consideration the amount of human and financial resources that are spent for its performance, especially for the first years of medical school. In this sense, an attempt to answer this question would be found if the grades obtained during medical school showed correlations between them and any relationship with passing the medical residency exam^12),(13.

Karay and Schauber reported relationships between the individual students’ development assessed by progress tests and performance in high stakes national licensing examinations, showing that the progress tests were suitable to monitor the students’ growth of knowledge during the course of medical training¹⁴.

A study performed at Universidade Federal de São Paulo searched for a predictable statistical model of success in medical residency exams based on the progress test and medical clerkship evaluations. Data on 114 medical students that took the progress tests and had clerkship evaluation at the 5^th and 6^th years were searched for predictors of the student’s score in the medical residency exam by linear regression analysis. The final model showed that the grades of the 5^th year clerkship rotations could predict the medical residency exam score, but grades of the progress test only approached the threshold of significance (p=0.06). Although the progress test failed to predict the final residency score, the univariate analysis pointed out to some correlation between the progress tests performed at the 4^th, 5^th and 6^th years¹⁵.

To investigate these correlations and eventually expand the number of factors influencing the residency exam, the factor analysis, which is a statistical method that searches for correlations between factors and allows the grouping of strongly correlated variables that explain a given phenomenon, would be considered appropriate¹⁶.

Thus, the objective of this study was to measure the correlation between grades on the progress tests, clerkship rotations assessments and medical residency exam and search for the strongest correlation with grades of the final medical residency exam using factor analysis.

METHODS

This was a retrospective and longitudinal study with correlation analyses of grades attained on the progress tests from the 1^st to 6^th year of medical school, the clerkship rotations performance coefficient (5^th and 6^th year) and medical residency exam in a cohort of students enrolled in the first year of medical school at the Universidade Federal de São Paulo in 2009.

The inclusion criteria were students enrolled in the 1^st year in 2009 at this Institution who undertook the progress tests (PT) from the 1^st to the 6^th year. The exclusion criteria were students who missed the progress tests at any time during the six years of the medical school.

Correlations between the grades on the progress tests, clerkship rotations coefficient and medical residency exam of all included students were performed using factor analysis^17),(18.

This study was approved by the institution’s Research Ethics Committee under number 2,555,803.

RESULTS

In 2009, 123 students were enrolled in the first year of medical school at the assessed University. Of these, 114 (92.7%) took the progress tests from 2009 to 2014 and had medical clerkship rotations evaluations (CAP 5^th and 6^th) in 2013 and 2014. Of the 114 students included in the study, 105 (92.1%) completed all stages of the residency exams at the University in 2015, and all 105 students passed the exam, thus being included in the analysis (Figure 1).

Figure 1 Flowchart of the study. PT: Progress tests at each year during the six years of Medical School; CAP: Coefficient of academic performance during Medical clerkship rotations; MR FR: Final result of Medical Residency Exam. 

Data reliability assessed by Cronbach’s alpha coefficient showed significant internal consistency between the grades on the progress tests from the 1^st to 6^th year, CAP 5^th year, CAP 6^th year and the medical residency exam, with a value of 0.736, 95% CI: 0.653 - 0.806 (p <0.001).

The average test scores are shown in Table 1. There was a progressive increase in the grades on the progress tests from the 1^st to the 6^th year of medical school, with statistically significant differences between all tests (p <0.001), except between the grades on the progress tests in the 3^rd and 4^th years, with a reduction in the grade in the 4^th year (p = 0.028). The average grade for the CAP 5th year was higher than the average for the CAP 6th year (p <0.001), and the grade on the residency exam was lower than the clerkship rotations grades.

The analysis of the communalities of the different exams varied from 0.46 to 0.839. The grades on the 1^st year progress test had a coefficient 0.46, suggesting less strength in the linear relationship with the other variables (<0.5) and were, therefore, removed from the factor analysis.

The analysis of the grades on the progress tests in the 2^nd, 3^rd, 4^th, 5^th and 6^th year, grades of the performance at the clerkship rotations (CAP 5^th, CAP 6^th) and grades on the medical residency exam regarding the Kaiser-Meyer-Olkin test (KMO) was 0.709 and the Bartlett’s sphericity test was significant (p <0.001), showing adequacy of the data for the factor analysis.

The extraction of factors, following the Kaiser rule, determined the extraction of three factors with an eigenvalue greater than 1 (Figure 2).

Figure 2 Scree Plot showing the three components with Eigenvalue higher than 1. 

The first component consisting of the variables CAP 5^th, CAP 6^th and medical residency grades explained 43.9% of the variance, whereas the second component was represented by the progress tests in the 5^th and 6^th year with 18.0%, and the third component, which consisted of the progress test grades in the 2^nd, 3^rd and 4^th years explained 14.4% of the variance of the model, totaling 76.3% of the total variance (Table 2).

The matrix rotation using the Varimax method allowed the visualization of the three components with greater correlation strength and generated a new correlation matrix. Component 1 was identified as CAP 5^th, CAP 6^th and medical residency grades, component 2 was formed by the grades on the progress tests in the 5^th and 6^th years and the third component comprised the grades on the progress tests in the 2^nd, 3^rd and 4^th years (Table 3).

DISCUSSION

The present study showed that, with some variations, the percentage of correct answers on the progress tests from the 1^st to the 6^th year of medical school and the medical residency exam scores at the Universidade Federal de São Paulo were similar to those of other medical schools^3),(9),(11.

Willoughby and Hutcheson²³ reported lower percentages of correct answers than those in the present study on the progress tests from the 1^st to the 6^th year, which were 6.1, 16.1, 30.7, 41.6, 50.9 and 56.0% at the University of Missouri. Likewise, Van der Vleutenet al³⁾ in a 15-year analysis of progress tests at Maastrich Medical School in the Netherlands, from 1977 to 1985, showed increasing percentages of correct answers from the 1^st to the 6^th year, with values ​​close to 20% in the 1^st year and 60% in the 6^th year. Another study also showed similar results on the progress tests for three classes of McMaster University students, with initial scores of 10 and 20% at the beginning of the course, with an almost linear rise until reaching 50% on the 5^th exam².

The theoretical medical residency exam score of 7.5 ± 0.6 was similar to that observed in another Brazilian University, in 2012 (7.6 ± 0.9), 2013 (7.3 ± 0.8), and 2014 (7.5 ± 0.9) in the multiple-choice questions in the areas of general surgery, internal medicine, gynecology and obstetrics, pediatrics and public health¹¹.

The factorial model obtained in the present study showed that although there were correlations between each of the tests, the strongest correlations were detected among three groups of students’ grades. The first group was represented by grades during the clerkship rotations and at the medical residency exam; the second group was represented by grades on the 5^th and 6^th year progress tests; and the third group by the 2^nd, 3^rd and 4^th-year progress tests.

Therefore, this analysis showed that the performance during the clerkship rotations, represented by the coefficients of academic performance during clerkship rotations and 5^th and 6^th year progress tests showed the greatest association with grades on the medical residency exam. The 2^nd, 3^rd and 4^th-year progress tests were also correlated with performance on the medical residency exam, although at a lower level.

Authors have reported an association between the global means of progress tests performed during medical course with the theoretical result of the medical residency exam, but not the final result of the medical residency exam. This fact could be explained because the progress test assesses the formative content, as well as the theoretical medical residency exam¹³. However, the present study aimed to search for factors correlated to the final result of medical residency exam, which measures a summative dimension. For this reason, the progress test grades, as well as the evaluation of the clerkship rotation had to be included in the factorial analysis, as clerkship evaluation represents a global knowledge^2),(24. According to the factor analysis, the final result of the medical residency exam was correlated with both the progress tests from the 2^nd to 6^th year and clerkship. The difference found between this study and the literature might be explained by the variables included in the analysis. As the final result of the medical residency exam included a summative evaluation of knowledge, it could explain the strong correlation with clerkship evaluation and the weaker correlation with the progress test².

After applying the factorial analysis technique, the 1^st year progress test was excluded from the analysis, possibly because of lower grades in the first year than in the subsequent years of the course, which resulted in a low correlation between them. Likewise, a study showed a significant Pearson’s correlation between the final result of the medical residency exam and the progress test scores, except for the 1^st, 2^nd and 3^rd years¹⁵. Other authors have already reported lower percentages of correct answers in the initial years with an almost linear rise towards the end of the course^2),(3),(23.

It should be noted that the reason why the 1^st year progress test was not included in the model was that its correlation was not as strong as the other tests and the fact that its communality was slightly lower than 0.5. Such technical issues of factor analysis are not absolutely rigid; however, it served to obtain a better model from the mathematical point of view. Thus, a more liberal approach in relation to the inclusion of the 1^st year progress test grades could have shown a correlation with less power in the explained variance and, perhaps, created a fourth component, with much less strength than the others. This analysis is in line with the idea that the progress test among first-year students has less discriminatory power because only a small percentage of the questions refer to the knowledge acquired at the beginning of the course³ and, furthermore, in the 1^st year of medical school the adhesion to progress might be lower than in more advanced years, with lower scores, which possibly reduced the strength of the correlation with later assessments²⁵.

Therefore, these results do not imply that grades of the 1^st year progress test do not provide any added value to the students’ performance on the medical residency exam. In this line, a study reported that initial levels and gains of knowledge during the medical course are related to later performances on national licensing examinations. The authors showed that the pre-clinical (1^st-4^th semesters) and the clinical phase (5^th-12^th semesters) of the medical course were both positively related to long-term assessments¹⁴.

Therefore, it can be said that the progress test would be important to understand the performance evaluation process of medical students at each year, to analyze the development of students within a particular cohort, so it could be used as a tool for early diagnosis and interventions, as well as for comparisons between medical schools^{1),(14),(24),(26}.

Therefore, considering all the research together, although in the present study, grades of the 1^st year progress test were not correlated with performance in other tests, students should be encouraged to take these tests in a committed way in the very beginning of the course. Moreover, efforts to improve the quality of the progress test and measures to improve students’ adhesion to it, especially in the first years, and better quality of the progress test would increase the correlation between these assessments^{2),(24),(25),(27}.

A limitation of this study was the analysis of only one group of students who started medical school in a given year. Additionally, except for the progress test, which could be interinstitutional, the inclusion of only one institution that may show clerkship rotations heterogeneity, different contexts of the practice of medicine and different forms of assessments compared to other schools of medicine, could represent an inclusion bias^28),(29, supporting the need for further studies.

The strength of this research was the longitudinal analysis of the test scores performed during the course, with great adherence by students to the progress tests and adequate reliability of data consistent with the internal consistency coefficient found in this study. Furthermore, the statistical treatment that was developed here allowed measuring the strength of the correlation factor, which might be in accordance with the daily experience. In addition, unlike other studies, the present research showed that the progress test applied in the last five years of medical school at our institution was also correlated with the final result of the medical residency exam, even though more studies are needed to validate these results.

CONCLUSIONS

This study showed that grades of the progress test from the 2^nd to 6^th year, clerkship rotations assessments and the final medical residency exam had strict and positive correlations between them. Furthermore, the academic performance during medical clerkship rotations showed the strongest correlations with the medical residency exam scores.