INTRODUCTION

Although scholars concur that longitudinal knowledge assessment is advantageous for medical education (^{Albanese & Case, 2016}; ^{Cecilio-Fernandes et al., 2021}; ^{Vleuten et al., 2018}; ^{Wrigley et al., 2012}), one of the great challenges facing current medical education is preparing assessments that can simultaneously i) measure the knowledge acquired by students throughout their studies (summative assessment), ii) be an instrument for the students to adjust their studies (formative assessment), and iii) help the professor and the educational institution to understand (informative assessment) and to improve knowledge acquisition (^{Pugh & Regehr, 2016}). One type of extensive assessment of cognitive knowledge that can perform these three functions, the Progress Test (PT), has been used in medical schools. It is periodically conducted and applied on the same day to all medical undergraduates, targeting the knowledge level expected for final-year students. However, there is uncertainty whether the current structure of PT is suitable to assess students in the early and intermediate stages of the program (^{Albanese & Case, 2016}; ^{Henning et al., 2017}).

PT arose in the wake of the constructivist pedagogy theory, which proposes as one of its principles that students are the builders of their own knowledge; the professor is no longer the granter of knowledge but a partner in the search for knowledge. This great change resulted in the use of new forms of approaching teaching and learning in medical education, called active methodologies, such as Problem-Based Learning (PBL), Team-Based Learning (TBL), etc. These new ways of teaching have generated the need to create new forms of assessing students and schools (^{Cecilio-Fernandes et al., 2021}; ^{Heeneman et al., 2017}; ^{Reberti et al., 2020}; ^{Vleuten & Schuwirth, 2019}).

Schools with traditional pedagogy, which do not use active methodologies, tend to assess student learning in each one of the disciplines or areas of knowledge separately. At the year-end exams, it is common for students to take many tests during the same week. They frequently study for each one of the tests at the last moment, adopting lower-order cognitive strategies such as cramming or memorizing by rote (^{Pugh & Regehr, 2016}). This kind of assessment promotes uncontextualized and fragmented learning that is forgotten after a short time. The PT, in contrast, because it covers a much broader content, is integrative and provides timely feedback, discourages the aforementioned strategy and stimulates a deeper, contextualized, and lasting knowledge (^{Vleuten et al., 2004}). Assessments with wide-ranging content that are systematically and regularly applied throughout the program allow students to verify their strengths and weaknesses of knowledge and potentially provide motivation and direction for future learning (^{Epstein, 2007}; ^{Heeneman et al., 2017}; ^{Tavakol & Dennick, 2017}).

Used in several countries since the 1970s, the PT is routinely applied in European and North American medical schools 2-4 times a year, with varying numbers of questions (^{Blake et al., 1996}; ^{Vleuten et al., 2004}). It emerged as an assessment more aligned with the problem-based curriculum and as a tool to compare the knowledge acquired using this type of curriculum with the traditional ones (^{Cecilio-Fernandes et al., 2021}). Currently, in the Netherlands, the test consists of 200 multiple-choice questions, with two-five options and penalties for wrong answers.

The items are built and revised by professors from different universities to avoid representing a vision and experience from an individual professor. In most cases, the questions are clinical vignettes, not long ones, that require a type of knowledge and logical reasoning involving higher-order cognitive processes. They are a matrix that comprises all knowledge areas expected by the end of the course. On a set date and time, all medical students take the same exam. Afterward, they receive their test scores individually, their acquired knowledge compared to previous year(s), and different ways of feedback.

In Brazil, since 2001, several consortiums of medical schools have been created to elaborate assessments with PTs. However, only one test of 120 simple multiple-choice questions (MCQs), with four answer options without penalization for wrong answers, is administered each year. With the support of the Brazilian Medical Education Association (ABEM), there are currently thirteen regional consortia that administer the PT in their medical schools. In 2015, 40% of the Brazilian medical schools had joined a PT consortium (^{Bicudo et al., 2019}; ^{Rosa et al., 2017}; ^{Sakai et al., 2008}; ^{Sartor et al., 2020}). The organization of schools in the form of consortiums for the PT promotes integration among schools, qualifies professors to elaborate items (questions), and reduces costs associated with the process of test elaboration, printing, and correction (^{Hamamoto & Bicudo, 2020b}; ^{Vleuten et al., 2018}).

Brazil has 357 medical schools, with 37,823 slots for new students in 2020; in 2001, 11,541 new slots were offered (^{Scheffer et al., 2020}). This sharp increase ignited the discussion about creating license exams for practicing medicine, in view of the need to create new external assessment formats to ensure the quality of schools and graduates (^{Bica & Kornis, 2020}; ^{Troncon, 2019}). The PT in Brazil is a form of longitudinal and essentially formative external assessment, offering different feedback opportunities. Regardless of the type of curriculum adopted by the school (traditional or PBL), the PT scores hardly ever affect regular student grades or whether the student will pass their year-end tests of final exams.

In some European and American medical schools, the PT results also add scores that will determine whether the student can progress through the program (summative assessment), or they are classified according to their performance on the exam (^{Albanese & Case, 2016}; ^{Blake et al., 1996}; ^{Heeneman et al., 2017}). These PTs must therefore be reliable (^{Downing, 2004}; ^{Kibble, 2017}). In Holland, four national PTs are administered each year to all medical students in order to increase assessment reliability (^{Wrigley et al., 2012}); this longitudinal knowledge assessment throughout the program also replaces the license exam at the end of medical programs (^{Vleuten et al., 2004}). However, these longitudinal, nationwide tests at such frequencies involve costs that many countries, such as Brazil, can hardly bear.

Both in Brazil and abroad, the schools receive their students’ general scores and the scores for the different knowledge areas covered in the medical curriculum, such as basic sciences, collective health, pediatrics, primary care, etc. By comparing students’ scores, both globally and for the different knowledge areas, with the mean for the group of PT-participant schools, it is possible for the schools to identify which areas correspond to their strengths and weaknesses. Based on this information, the schools can alter their curriculum, processes and teaching and learning strategies so that in the following years they can evaluate the impact of these changes (^{Rosa et al., 2017}).

PT organizers, in all parts of the world where it is applied, after the exam is employed, analyze the test’s validity and reliability, its pedagogical aspects, and the psychometrics of each item. They select some questions and modify others to build a question database. The emergence of computerized adaptive testing made it possible for questions to become more or less difficult as the student answers the test, depending on their skills and knowledge. The existence of an item bank, with questions previously tested and calibrated, is fundamental for this form of assessment. An international computerized PT is already available and can provide results with a high level of reliability, using a smaller number of questions (^{Cecilio-Fernandes, 2019}; ^{Collares & Cecilio-Fernandes, 2019}).

In Brazil, the Ministry of Education has been applying a nationwide exam, the “Exame Nacional de Desempenho dos Estudantes (ENADE)”, to all final-year undergraduates, including medical students, since 1996. This exam was changed in 2004 to include first-year students in order to determine programs’ contribution to student learning. However, after ENEM - the exam applied to senior high school students and used for admission to public universities and as a quality reference for a number of public higher education inclusion policies - became national in scope, this initiative was abandoned, and now only graduating students are required to take it. Since this forty-question exam (with only thirty questions directly addressing the professional field) is applied to each career only once every three years, it has little formative impact and is clearly inappropriate for assessing students’ learning progress (^{Ristoff, 2022}). This has led medical schools to move forward with other forms of assessment such as the PT.

To improve the reliability of PTs and to overcome the economic issues related to running multiple tests each year, the development of a better-quality PT is a promising approach to improve the assessment of medical students’ knowledge (^{Aubin et al., 2020}; ^{Sahoo & Singh, 2017}). Therefore, this study aimed to analyze the reliability of PTs administered in Brazil for students at different stages of their studies (2nd, 4th and 6th years). Based on our results, we propose a new format of longitudinal knowledge assessment, called the Customized Progress Test (CPT), which can evaluate medical schools and the progress of students throughout their studies, serving as an alternative to license exams applied only at the end of programs.

OBJECTIVES

METHODS

RESULTS

The number of students participating in the PTs (2nd, 4th and 6th-year students) and the average scores they obtained in the three analyzed years (2015, 2016, and 2017), are shown in Table 1.

The students’ mean scores progressively increased from the 2nd year (41.8) to the 4th year (51.8) and to the 6th year (60.5); the differences among averages were statistically significant.

The items’ difficulty and discrimination mean values are shown in Figure 1.

Source: The authors.

FIGURE 1 Mean difficulty and discrimination values for PT items measured for students in different program years - means for 2015, 2016 and 2017 - ten medical schools in Brazil’s South Region 

The percentage of items considered difficult decreased from 60.9% (for 2nd-year students) to 42.1% and 25.3% (for 4th and 6th-year students, respectively), whereas that of questions considered easy increased from 5.3% (for 2nd-year students) to 12.1% and 21.1% (for 4th and 6th-year students, respectively) (Figure 1A).

The percentage of items with poor discrimination (bad quality) decreased from 2nd-year students (76.4%) to 4th and 6th-year ones (47.7% and 25.3%, respectively). All these comparisons between question characteristics for students in different program years proved to be significant (p < 0.001) by ANOVA (Figure 1B).

The reliability of the three PTs for the three program years and the years they were administered are shown in Table 2.

The mean reliability of the three PTs, measured by Cronbach’s alpha, increased from 2nd-year students (0.60) to 4th and 6th-year ones (0.76 and 0.87, respectively). The differences between the mean values were significant (p < 0.001), and there was no overlap of confidence intervals for the mean values in any of the comparisons of PTs’ reliabilities between students in different program years.

DISCUSSION

The mean score differences between 2nd, 4th, and 6th-year students, in all PTs described in Table 1, show a similar increase in the three years analyzed and are in accordance with the national (^{Bicudo et al., 2019}) and international (^{Wrigley et al., 2012}) literature.

In order to contribute to understanding the current PTs’ assessment approach, this study evaluated i) item difficulty, ii) item discrimination index, and iii) test reliability of PTs administered to 2nd, 4th and 6th-years students from 10 Brazilian medical schools in 2015, 2016, and 2017. The percentage of items considered difficult decreased, and those considered easy increased from the 2nd to the 4th and 6th years (Figure 1A). The discrimination index value for questions considered poor decreased, and for those considered acceptable or good, it increased from the 2nd to the 4th and 6th years (Figure 1B). The discrimination index determined through point-biserial correlation is a good indicator of an item’s quality (^{Tavakol & Dennick, 2017}). Point-biserial values below 0.2 suggest that the question is not able to discriminate between students with different levels of knowledge, and thus must be considered cautiously (^{De Champlain, 2010}; ^{Kibble, 2017}; ^{Pasquali, 2004}; ^{Tavakol & Dennick, 2013}). The 76.4% percentage of poor-quality items (point-biserial < 0.2) for 2nd-year students, found in this study, probably occurs because the PT is usually designed to target the knowledge level expected for final-year students, influencing the test´s reliability.

The reliability of PTs is influenced by the quality and number of questions and how frequently the test is administered (^{Kibble, 2017}; ^{Wrigley et al., 2012}). The mean reliability of the PTs in this study (Table 2), measured by Cronbach’s alpha, was 0.60 for 2nd-year students, which is considered low and unacceptable for moderate or even low-stakes assessments (^{Downing, 2004}; ^{Kibble, 2017}). The mean reliability increased to 0.76 for 4th-year students, a number considered reasonable, and to 0.87 for 6th-year students, a number considered close to ideal. This high reliability for the 6th year suggests that the PT exams are reaching the expected consistency for this kind of assessment for the students in the final year of the medical program (^{Downing, 2004}; ^{Kibble, 2017}; ^{Pasquali, 2004}). In a previously described PT study (^{Wrigley et al., 2012}), similar values for 2nd and 4th-year students were observed. In the present study, the PTs’ reliability of 0.60 for 2nd-year students indicates that the PT scores presented a larger random error for this program year (^{Tavakol & Dennick, 2011}). Such a high number of errors measured in a single annual PT with 120 MCQs, as in this study, makes the test´s usefulness impractical for the current assessment of 2nd-year medical students in Brazil (^{Albanese & Case, 2016}; ^{Downing, 2004}; ^{Kibble, 2017}). Thus, this low reliability confirms the difficulty of using this format of PT in Brazil as a longitudinal assessment aimed as an alternative to a national license exam for professional medical practice.

The standard performance values used in this study for item difficulty, item discrimination index, and test reliability can be questioned. Some authors consider items with less than 20% (^{Aubin et al., 2020}) or 30% (^{De Champlain, 2010}; ^{Kheyami et al., 2018}; ^{Primi, 2012}; ^{Sahoo & Singh, 2017}) correct answers as difficult. These values are normally used when there is a penalty for incorrect answers (formula scoring), in order to avoid guessing. Regarding the discrimination index, some authors consider items with a point-biserial correlation coefficient of 0.25 or above (^{Tavakol & Dennick, 2017}) as good. However, it is almost unanimous that a test with a reliability value below 0.70, as observed for 2nd-year students in this study, cannot be used to assess the academic development of students.

In some European locations, where PT emerged over fifty years ago, the purpose was to evaluate curriculum changes and create examinations aligned with PBL curriculums. Interestingly, it was initially used only as a formative assessment. Gradually, it substituted the traditional forms of summative assessment. With the increase in the frequency of annual examinations in order to increase the reliability of results, in some places, the PT became the only tool to evaluate knowledge or its acquisition. The PT has been consistently one of the components of the assessment system, which also analyzes skills and competencies. However, this is not currently used in Brazil, where the PT started to be employed less than twenty years ago and is mainly used as a formative tool.

The discussion about the need to use one PT each semester (twice a year) and about ways of integrating the results into summative assessments has begun in our consortiums. However, both in schools with modern curriculums, pedagogically aligned with the PT, and in schools with traditional curriculums, this discussion needs to be deepened. Moreover, it seems necessary to increase student adherence and commitment, especially since the initial phases, and to avoid guessing behavior. However, this issue has not been studied or described in the literature regarding PT and medical education in Brazil.

In general, the PTs are developed and employed by a group of schools. This collective initiative reduces the costs of PT employment and analysis and increases the quality of questions. In addition to the advantage for students, since they can rectify their journey early on, the PT also provides a comparative parameter to the institutions, which can analyze their strengths and weaknesses. In Brazil, the concern with results and their use as a tool for changes in schools seems to elicit more interest from private schools than public ones, as apparently occurs already with ENADE (^{Damas & Miranda, 2019}).

In order to avoid the risks of ENADE, which creates a ranking and a rivalry for positions between schools and might serve as a parameter that can cover up issues, with PT, each school receives only its own results, and the average results for the group of participant schools serves as a comparison factor. In addition to fostering integration between the schools in the consortium, this environment of collaboration strengthens ABEM regional branches. Moreover, it stimulates the creation of an assessment research area and holds the potential to improve knowledge assessments and medical education (^{Hamamoto & Bicudo, 2020a}).

The elaboration of tests with well-structured multiple-choice questions is considered adequate to assess the theoretical knowledge evolution of students, also called cognitive domain assessment (^{Aubin et al., 2020}; ^{Epstein, 2007}; ^{Vleuten & Schuwirth, 2019}). This must partially explain why the PT is being increasingly applied in more countries (^{Vleuten et al., 2018}) and in different undergraduate programs, such as dentistry (^{Ali et al., 2016}; ^{Oliveira et al., 2020}), pharmacy (^{Albekairy et al., 2021}) and veterinary (^{Herrmann et al., 2020}), as well as in postgraduate medical training (^{Rutgers et al., 2018}), both in Brazil and abroad (^{Alkhalaf et al., 2021}; ^{Sá et al., 2021}).

Because of its wide content, the rise of PT discourages last-minute study and encourages continuous study, thus representing an advancement in educational assessments in the healthcare field (^{Pugh & Regehr, 2016}). However, the most modern concepts of assessments suggest that each school should create not only one or two types of evaluation, but assessment systems that cover all theoretical knowledge domains, skills, and attitudes (^{Norcini et al., 2018}). The same concepts should be applied to large-scale assessments.

Other essential domains for professional practice, such as communication skills, clinical skills, and attitudes should also be evaluated (^{Epstein, 2007}; ^{Vleuten, 2016}). However, they require much more complex, laborious, and expensive evaluation formats, such as Objective Structured Clinical Examination (OSCE), Mini Clinical Evaluation Exercise (Mini-Cex), 360 Degree Feedback, etc. The application of these methodologies in large-scale external educational assessments deserves further study and analysis to become viable for use in Brazil.

If one wide, large-scale longitudinal assessment system is inadequate for determining whether a professional has enough knowledge and is able to practice medicine, even more so would be a single theoretical exam, such as the professional license exam, applied at the end of medical programs. Such an exam would neither directly impact schools nor improve medical education. It would be applied to test students, whether or not they acquired theoretical knowledge during their undergraduate studies, without having allowed them the possibility of remediation while in the program. Hence the need for reliable longitudinal assessments for students at different stages of the program.

Thus, considering only the assessment of theoretical knowledge, this study supports the perception that the current PT format, i.e., 120 MCQs, with a test administered once a year in Brazil, targeting the knowledge expected for 6th-year students, is unsuitable for assessing the knowledge acquired by medical students since the initial medical school years. The PT can and should be improved for students in their initial and intermediate stages.

This study is the first of a series that aims to investigate and propose a reliable longitudinal assessment that measures knowledge acquisition by students in all their study years in the medical program. Thus, we intend to apply a new type of PT called the Customized Progress Test (CPT). In this approach, the items for 2nd-year students will be divided as follows: 25% will cover the expected knowledge level for their year, 25% for the 4th-year level, and 50% for the 6th-year level; for 4th-year students, 50% of the questions will cover the expected knowledge level for their year and 50% for 6th-year level; and for 6th-year students, all questions will cover the expected knowledge level for their year, as is currently done with excellent reliability (^{Wrigley et al., 2012}). It is expected that this approach will help to provide an assessment (1) with good reliability for all stages of the program and (2) which can be used as an evaluation with moderate-high stakes for examinees.

In its current state, the yearly longitudinal PT in medical programs is unreliable for evaluating the knowledge that medical students acquired over their medical program years. We hope that this study will reinforce the relevance of proposing the Customized Progress Test as an innovative strategy for formative and summative assessments, potentially contributing to significantly improve the evaluation of medical education in Brazil. Such an assessment as the one proposed by the CPT has the potential to be a good alternative to the proposed national licensing examination at the end of healthcare programs.