text in 
English (pdf)
Article in xml format
Automatic translation
Send this article by e-mail
Permalink
INTRODUCTION
Program evaluation is the process of obtaining information about a teaching course or program that takes into account aspects of cost-effectiveness, checking the adequacy of the evaluation according to the course purpose and the program’s capacity to generate changes in real life. This assessment provides information that is analyzed for the creation of reports, the issuing of value judgments and decision making aiming to qualify the training of future professionals who will be delivered to society, including health professionals1. An educational program itself is rarely static, so an assessment plan should be designed to allow educators to obtain useful knowledge about the program and to support its continuing development2. Such regular assessments provide feedback for decision-making that aim at better teaching and learning practices1.
Among the program evaluation models is the four-level Kirkpatrick model3, which is based on the assumption of linear relationships between the program components and the results, being useful to help evaluators identify students’ relevant results. There is also the Logic Model4, which specifies the intended relationships between its evaluation components and may require constant updating as the program evolves. There’s the CIPP model from Stufflebeam5, flexible enough to incorporate studies that support continuous program improvement, as well as summative studies of the results of a completed program6. Another model used to plan and carry out program evaluation is the task-oriented one, which has five steps with a proposal guided by guiding questions that need to be answered during the evaluation process; this model is quite simple and self-explanatory1.
The Mini Clinical Evaluation Exercise (MiniCex), is a performance rating scale developed by the American Board of Internal Medicine in the 1990s, designed to assess the skills that medical students and residents need in situation of real-life with patients7,8.
There is no description in the literature on the use of MiniCex as a source of information for a program evaluation in undergraduate medical students; however, due to the quality of the data that can be obtained through the use of this instrument, it was considered that it could be a relevant source of information about the skills acquired throughout the period prior to the internship, which might even be used to guide the activities during the internship.
Therefore, the objective was to use the MiniCex as part of a program evaluation at the beginning of the medical school internship, aiming to determine the areas of the basic and pre-clinical course in which the students have difficulties.
METHOD
A cross-sectional, descriptive study was carried out using retrospective data obtained through MiniCex forms applied to students in the ninth semester in the Clinical Medicine I module, which corresponds to the first semester of the internship at the Medical School of Federal University of Pará (UFPA), with a total of 111 students being evaluated among the 154 eligible for internship from August 2017 to July 2018.
Only the first assessment obtained through the MiniCex of each student was used for this research, considering that it would represent the skills acquired in the previous eight semesters.
Clinical Medicine I consisted of outpatient activities in clinical medicine, cardiology, rheumatology, hepatology, geriatrics and pneumology, carried out by students in the ninth semester of the medical course.
The instrument used by the institution evaluated the six traditionally observed domains of clinical interview skill, physical examination skill, humanistic quality/professionalism, reasoning and clinical judgment, communication and counseling skills, organization and efficiency, ending with overall clinical skill), which were classified using a 6-item Likert scale: 1 to 3 - unsatisfactory and 4 to 6 - satisfactory. Other data were recorded, such as: case complexity, patient diagnosis, observation environment, whether it was a new case or return, as well as activity module.
The students were instructed on the first day of activities about the instrument use, and both the student and the observer were given the opportunity to identify evaluation opportunities. Feedback was regularly offered immediately after the teacher’s observation.
Teachers were trained in workshops carried out by the institution, both in the use of MiniCex and in providing feedback. The forms were kept by the teachers, which were requested at the end of each module for data collection and to prepare reports that were sent to the course management by the researchers.
The study was approved by the Research Ethics Committee of Instituto de Ciências da Saúde da UFPA, under number 2,250,664 (CAAE 66666017.9.0000.5172) on 08/31/2017. The collected data were organized and analyzed using Microsoft Excel 2007® spreadsheets. Categorical variables were expressed as absolute and percentage values, using the chi-square test to assess differences between the groups. A p value <0.05 was considered statistically significant.
RESULTS
Throughout the 12-month project development, a total of 111 students from the ninth semester of medical school, attending Clinical Medicine I internship, were evaluated through the MiniCex scale, of which observations were made by 4 teachers. Among the performed evaluations, of which 97.2% were requested by the teachers, the majority (72%) were of new cases, while 45% and 38.7% of them showed low and moderate complexity, respectively (Table 1).
Table 1 Data obtained through MiniCex on the type and complexity of cases treated by the internship students in Clinical Medicine I, from August 2017 to July 2018.
| New case or return case? | Frequency | Percentage |
|---|---|---|
| New case | 80 | 72% |
| Return | 25 | 22.5% |
| No information | 6 | 5.4% |
| Total | 111 | 100% |
| Complexity | ||
| Low | 50 | 45% |
| Moderate | 43 | 38.7% |
| High | 5 | 4.5% |
| No information | 13 | 11.7% |
| Total | 111 | 100% |
Source: MiniCex forms
The diagnoses identified during the care assistance corresponded to 52 diseases, grouped into 11 systems. There was a predominance of the musculoskeletal system with 20.7%, followed by the gastrointestinal / hepatology system with 11.1% (Table 2). As for the skills in each domain, the performance was satisfactory in all of them (Table 3). In this project, it was observed that 12.6% of students had difficulty in at least 1 area, followed by 6.3% of students with difficulties in 2 areas and 4.5% with unsatisfactory performance in 3 or more areas.
Table 2 System frequencies in Clinical Medicine I services obtained through Mini-Cex applied to internship students from August 2017 to July 2018.
| System | Frequency | Percentage |
|---|---|---|
| Musculoskeletal | 28 | 20.7% |
| Gastrointestinal/Hepatology | 15 | 11.1% |
| Neurological | 14 | 10.3% |
| Cardiovascular | 10 | 7.4% |
| Endocrinological | 9 | 6.6% |
| Skin | 6 | 4.4% |
| Respiratory | 6 | 4.4% |
| Nephrological/Urological | 5 | 3.7% |
| Reproductive | 3 | 2.2% |
| Hematological | 3 | 2.2% |
| General | 2 | 1.48% |
| No information | 34 | 25.1% |
| Total | 135 | 100% |
Source: MiniCex forms
Table 3 Performance of students in the ninth semester of internship, assessed through MiniCex during Clinical Medicine I, from August 2017 to July 2018.
| Insufficient | Sufficient | |||
|---|---|---|---|---|
| Frequency | Percentage | Frequency | Percentage | |
| Clinical interview skill | 14 | 13,8% | 87 | 86,1% |
| Physical examination skill | 15 | 28,3% | 38 | 71,6% |
| Reasoning and clinical judgment | 9 | 13,1% | 60 | 86,9% |
| Humanistic quality / professionalism | 0 | 0% | 55 | 100% |
| Communication and counseling skills | 1 | 1,7% | 56 | 98,2% |
| Organization and efficiency | 5 | 10% | 45 | 90% |
| Overall clinical skill | 0 | 0% | 26 | 100% |
Source: MiniCex forms
DISCUSSION
This study showed that the information obtained from the use of MiniCex can be used as part of a program evaluation during the medical course.
The internship students demonstrated sufficient overall competence in all skills in the Clinical Medicine module, highlighting the “humanistic / professionalism skill”, in which 100% of the students showed satisfactory performance, corroborating the study by Baños (2015)9, in which the students had a satisfactory result and had a higher score in the same area. The project showed that 12.6% of students had difficulty in at least 1 area, followed by 6.3% of students with difficulties in 2 areas and 4.5% with an unsatisfactory performance in 3 or more areas. There are no previous studies in the literature that allow a comparison with our findings.
Even with the lack of literature for comparison, it is noteworthy that a feedback to teachers of the preclinical stages responsible for teaching skills, showing the totality of observations made to assess the “humanistic quality / professionalism” skill were reported as satisfactory by the internship teachers, which would reinforce that they are on the right path regarding this skill. On the other hand, going back to these teachers with the information that almost 30% of the students evaluated at the beginning of the internship were considered as showing insufficient capacity in the “physical examination skill”, points to the need to review their programs and / or methodologies, aiming at an improvement in this performance. It is precisely this type of information that the MiniCex is able to offer as part of a program evaluation, objectively pointing out the skills that can be improved and those that are at an appropriate level.
Other assessment instruments could have been used, such as the Objective Structured Clinical Examination (OSCE), considered one of the most reliable methods for assessing the clinical skills of students and residents, in which the examinees alternate through a certain number of stations where there are real or standardized patients, with the purpose of performing different clinical tasks10, as well as the Problem Based Learning (PBL), a method that allows the student to experience learning by doing and, especially, the direct involvement with real life and the clinical environment 11.
However, in this study, MiniCex was chosen because it is a practical instrument developed to be applied in approximately 20 minutes 8. The MiniCex assessment involves direct observation by an educational supervisor of an student’s performance in real clinical situations. The evaluation is repeated on several occasions, and can occur in several situations: clinical, on-call, surgeries, etc. with subsequent feedback, which should be given as soon as possible 12. The main characteristic of this formative assessment instrument for clinical skills is to reproduce as closely as possible the routine of attending physicians in their workplace. It is precisely in the way of dealing with sensitive situations that the interns will show their skills, because in professional life, they won’t be examining actors or dummies. The students must learn to make decisions under conditions of uncertainty, to deal with ambiguity, complexity, exceptionality and the conflicts of values that almost always escape technical reasoning13. In parallel, as a disadvantage, the instrument requires more than one meeting between the student and the patients in order result in a more reliable and valid measure of practice and development of clinical skills 8. This study was limited to presenting the data from the first MiniCex form of each evaluated student, as we aimed to use the instrument as part of a program evaluation. However, it is worth mentioning that other MiniCex forms were filled out and used also for student evaluation, with their respective feedback, according to the opportunities during the Clinical Medicine activities.
The process of evaluating educational programs is the “systematic collection and analysis of information related to the design, implementation and results of a program, aiming at monitoring and improving the quality and effectiveness of the program” 14. The initial evaluation phase comprise the moment when the institutions or individuals responsible for a program make the decision to evaluate it. They must decide about the objective (s) of the evaluation and who will be responsible for carrying it out.2 Medical educators can choose between the models of individual program evaluation instruments or a combination of them to develop an appropriate evaluation model for their programs 6.
Special attention should be paid to the research objectives, the method’s validity and the selected instruments. The assessment can have a formative role, identifying areas where teaching can be improved or a cumulative role, assessing the effectiveness of teaching 2. As important as evaluating a program, is to use the instrument to give feedback to the student evaluated via MiniCex as one of the educational and evaluation strategies with the greatest evidence of effectiveness in the education of health professionals15, as it is an important part of the process of improvement of clinical skills 16, as well as professional development 17. As a result, the evaluation becomes a regulatory activity in the teaching-learning process, detecting gaps and providing solutions to any obstacles faced by students, in addition to providing improvements in didactic tools and possible adjustments in the syllabus or even in the curricular structure. The feedback regulates the teaching-learning process, continuously providing information so that the students can perceive how distant, or close, they are from the desired goals. The fact that the feedback is continuous allows the necessary adjustments for the best quality of learning to be made early, and not only when the student fails the test at the end of the course, that is, in the summative evaluation 18. Providing good quality and timely feedback plays an essential role in learning and professional development in medicine19.
In this context, the act of evaluating the professional in training starts to acquire a new meaning: it stops establishing whether the student has had a passing grade or not in a certain discipline to check whether the educational objectives have been achieved as a whole; whether the student has acquired, in addition to the necessary technical knowledge, competencies, skills and attitudes required for the required new professional profile. Therefore, it divests itself of the summative characteristic alone and assumes a formative role, as an integral part of the new professional’s training skills 20.
Considering the concerns that the undergraduate students are not adequately prepared for the practice and do not meet the national health needs, evidence is needed to indicate what is happening, where improvements are needed and how to make these improvements. Therefore, it is necessary to identify a system for the continuous screening of the program deficiencies and its possible improvements 21.
FINAL CONSIDERATIONS
The MiniCex applied to medical internship students showed to be capable of providing important information, useful as part of a program evaluation of the areas prior to internship. The analysis of the obtained data was sent to teachers of the pre-internship, internship school and course management. To the first ones, with the objective of reviewing their programs, detecting where they can intervene and, thus, making changes that aim at better acquisition of basic knowledge by students and, consequently, improve their performance. To the second ones, aiming to give them an overview of where they will need to focus their programs according to the needs identified in the students who arrive at the internship phase. Finally, to the management, as a guide of what should be supervised by the professionals teaching the semesters that precede the internship.
