INTRODUCTION
The teaching of surgical techniques in medical residency programs has historically followed Halstead’s model of “see one, do one, teach one”1. It is a model based on surgical experience, in which proficiency is determined by the number of surgeries performed in key procedures2, with little emphasis on details or technical skills3.
Although this teaching-learning model is widely prevalent in Brazil and most of the world, the relationship between proficiency and experience is controversial, reinforced by studies that questioned the usefulness of the number of surgeries performed as a competency marker4. Some people require less exposure to a series of challenges to be able to assimilate the necessary skills for good surgical performance, while others require a longer training time5.
Some factors have been pressing for changes, such as greater concern about ethics and patient safety, the demand from society for greater efficiency and quality of health services and the limitation of the resident’s workload in some countries3),(5),(6.
In this scenario, the teaching of the surgical technique has been widely discussed, undergoing significant changes, especially in some developed countries. The teaching model based on the acquisition of competencies is gradually replacing the traditional Halstead’s model. In this new model, students cannot progress and cannot complete their training if they do not demonstrate competency in predefined areas of the curriculum7.
The demonstration of competency in the context of surgical training requires the assessment of several attributes, such as knowledge, technical skills, integrity, communication, leadership and teamwork8),(9. It must, therefore, be inferred from the performance analysis, that is, the way the person performs and behaves in daily practice10.
One class of competency-based assessment instruments has emerged in the field of specialized medical training: the Workplace-Based Assessment (WBA). Aiming to assess activities routinely performed in the work environment, the WBA offers several advantages, such as the possibility of focusing on a particular area of education or skill, better measurement of its own progression and identification of the resident with low performance, allowing better planning of measures that can correct the course of learning11.
One of the most relevant instruments belonging to this class is the Procedure Based Assessment (PBA), widely used in the United Kingdom, where, since 2007, it has been adopted as the main method for assessing surgical skills12.
In the PBA, the resident is directly observed during the surgical procedure, being evaluated in relation to a checklist of competencies and receiving immediate feedback.
The PBA is designed to support the development of technical and professional skills, incorporating constructive feedback, allowing the assessment of skills during professional practice13. Due to these characteristics, it can fulfill a dual purpose as an assessment method: summative and formative14.
This study aims to present the initial experience in the creation and implementation of the PBA in a Medical Residency Program in Urology in Brazil.
MATERIALS AND METHODS
The study was approved by the Research Ethics Committee, under CAAE number 16950719.0.0000.5292, Opinion N. 3.532.085.
Between July 2019 and July 2020, a prospective, exploratory, action-research study was developed, involving 10 preceptors and six residents, with two residents from each calendar year, from the Medical Residency Program in Urology at Hospital Universitário Onofre Lopes (PRMUro-HUOL).
The primary objective was the development of six PBA protocols per group consensus. The secondary objectives were team training and performance analysis based on the application of the PBA. For that purpose, the study was developed in two main phases: 1- group training and preparation of PBA protocols; and 2- PBA implementation.
Training of the participants
It occurred in two phases, the first being aimed at the presentation of relevant content for performance assessment and feedback and the second aimed at the supervised application of the PBAs. Both were carried out in-person and integrated the preceptors and residents of PRMUro-HUOL.
The first phase lasted four hours and addressed issues related to student assessment, with emphasis on assessment methods in medical practice, the PBA and the importance of feedback. The methodology used was the video-mediated teaching, by presenting real situations for evaluation, discussion in a conversation circle format, exploring the participants’ previous experiences, as well as a brief dialogue exposition for introductions and closings of the addressed contents.
The PBA protocols were applied at the second phase of the training, under the supervision of two of the authors, who acted as instructors. The objectives of this phase were to promote familiarization with the PBA, resolve doubts and improve the application of the method. In this phase, 10 PBAs were applied during the daily residency activities, as shown in Figure 1.
Creation of the PBA protocols
The creation of the PBA protocols was carried out using the methodology of group consensus, interspersing rounds of in-person and remote interaction and feedback, which are summarized in Figure 2.
The steps of this process were the following:
Definition, by the authors, of the six most relevant urological surgical procedures in the Residency Program, with two being related to each calendar year. The procedures chosen were: rigid ureterolithotripsy and open prostatectomy, related to the first year of residency; transurethral resection of the prostate and laparoscopic nephrectomy, corresponding to the second year of residency; and percutaneous nephrolithotripsy and laparoscopic radical prostatectomy, related to the third year of residency.
Creation of six pilot PBAs by the authors, based on existing protocols abroad. Of the six protocols, four had a model in a foreign language and two were originally developed by the authors.
Division of the expert group into six subgroups. Each subgroup developed a PBA model based on the pilot PBA within a one-week period and made it available for full group evaluation. In addition to the specialists, each subgroup had a resident.
Through the Google form application, each subgroup was able to give their opinion and suggest changes to the protocols developed by the other subgroups, for a period of one week;
Implementation of changes by the authors, resulting in new versions of the protocols, which were presented to the participants.
Application of the PBA in practice, under the supervision of two of the authors.
Final round of in-person interaction and feedback, with all participants, after the supervised application of the PBAs. Additional adjustments were discussed and included in the protocols, concluding the process.
Main characteristics of the developed PBAs
Division of the competency checklist into four main domains: preoperative planning, preoperative preparation, intraoperative technique and postoperative care;
The evaluation options for each item in the competency checklist are: “does not apply”, “insufficient”, “good/regular” and “excellent”;
After the competency checklist, a “general assessment” grade should be assigned, with the options: “insufficient”, “good/regular” and “excellent”;
The instrument is finalized with the “Assessment of the Competency Level”, a reference to the assessment of the Intercollegiate Surgical Curriculum Program (ISCP - United Kingdom) called “Global Summary Score”, a progressive scale related to the level of autonomy of procedure performance by the resident.
As an example, the PBA protocol for laparoscopic radical prostatectomy is shown in Figure 3. The other five PBA protocols can be accessed online at https://doi.org/10.6084/m9.figshare.c.5560734.v1.
PBA implementation
The implementation of the PBAs occurred in two phases: in the training of the participants, during which this activity was supervised by the authors; and after the training, in an unsupervised manner.
Analysis of performance results in the evaluations
In addition to the descriptive analysis of the data, the analysis of the residents’ performance was carried out based on the assessment of the competency checklist. The calculation of the Excellent General Score (ExcGS) was agreed upon, which corresponds to the percentage of items scored as “excellent” among those that were assessed.
In this data analysis, we sought to assess the correlation between the performance measured in the PBA and the training time to verify whether the residents showed a measurable improvement in performance by the PBA over time, that is, as they accumulated more time of training, considering the 148 days elapsed between the first and the last assessment.
Data were analyzed in an Excel 2016 spreadsheet and recorded in scatter plots (Graph 1). The intensity of the correlation between the variables was determined by calculating Spearman’s coefficient.
RESULTS
All residents and 77% of the program’s preceptors (n=10) were trained to use the six PBAs, all men. Among the preceptors, 40% had a Master’s Degree, but none reported a previous experience with the PBA.
In total, 31 PBAs were applied, 10 during the training phase and 21 after it. The time elapsed between the first and the last assessment was 148 days. All residents were assessed, with four to six assessments being carried out per resident, an average of 5.2 PBAs/resident.
A positive correlation was demonstrated between training time and PBA performance in the laparoscopic radical prostatectomy, percutaneous nephrolithotripsy, laparoscopic nephrectomy and in the set of the six procedures, with a Spearman’s coefficient of 0.97, 0.55, 0.42 and 0.31, respectively. A trend line was included in the scatter plots of these procedures (Graph 1), for better visualization.
DISCUSSION
The modernization of surgical training with the introduction of the teaching model based on the acquisition of competencies necessitated the development and implementation of mechanisms for monitoring and evaluating the intended competencies15.
Occupying a central space in this process, WBAs are designed to integrate teaching, learning, assessment and feedback, assessing skills and behaviors, rather than just knowledge16.
The use of WBAs that address surgical competencies has been shown to be quite limited in medical residencies in Brazil, with no consistent publications on the subject.
In addition to not experiencing an environment of intense debate on the surgical teaching model, other factors can negatively impact the change initiatives in Brazil. The scarcity of assessment tools validated in Portuguese and the difficulty of accessing instruments in other languages stand out.
In this study, we describe the first experience with the use of the PBA - a tool for assessing surgical skills in the workplace - in a Medical Residency Program in Urology in Brazil.
Using a group consensus methodology, all participants had the opportunity to contribute and give their opinion on equal terms, based on several rounds of interaction and feedback, building protocols that were finalized with the group’s approval.
Training is an important step in this process, as it improves the group’s understanding of the teaching-learning process and student assessment concepts, as well as increasing the efficiency and quality of the instrument’s application.
The strategy of creating the instruments per group consensus, including the aggregation of the Service’s residents in the process, together with the training process, proved to be a good option for the introduction of this assessment tool. In addition to the aforementioned benefits, this strategy reflected the concern to create an environment of team integration and engagement around a completely new activity in the routine of PRMUro-HUOL. This is a particularly important point, because the lack of team engagement is identified as one of the main factors of failure in the use of WBAs16),(17.
When applying PBAs, there was a trend towards improvement in the performance of the PBA in three of the six surgical procedures and in the joint assessment of the six procedures, considering the approximate interval of five months between the first and last assessments.
This finding is in agreement with the literature, which demonstrated that PBA scores are related to surgical training time12 and that the sequential use of the PBA should demonstrate progression towards competency13.
The large-scale use of the PBA in the United Kingdom allowed the validation of this tool in different studies, which also reported the reliability of the method12),(13. Another evaluated aspect was feasibility and, in this sense, it was concluded that the PBA is viable and valued by students as a means of constructive feedback and training direction18.
Study limitations
This study was negatively impacted by the COVID-19 pandemic, due to the limited scheduling of elective surgeries in certain periods of the study, resulting in a smaller number of PBA applications than initially planned. This made it difficult to obtain more significant results in relation to the performance analysis.
CONCLUSIONS
The creation and implementation of assessment protocols based on the PBA from a consensus of a group of experts is feasible and is in line with the most modern concepts of the surgical training model based on competency acquisition.
The PBA is an important part of this training model in some countries, and it can be validated in the national territory as other Residency Programs start to adopt them.