INTRODUCTION
The current approach to health education is based on the competencies that will be developed during the physician and the specialist’s training. Competency can be defined as the correlation between the use of knowledge, clinical and technical skills, communication skills and reasoning, reflection and professionalism in daily clinical practice for the benefit of people or the community1.
In Brazil, medical training follows the guidelines of the National Curriculum Guidelines (DCNs), which incorporate the most modern concepts of adult education, and establish the general and specific competencies to be developed during training, based on the desired profile of the recently graduated students2.
Similarly, also in medical residency, competencies continue to be developed and evaluated. According to the National Commission of Medical Residency (CNRM), the Resident Physician will be periodically evaluated in their training through written, oral, practical or performance tests, and also through attitude scales, which include attributes such as: ethical behavior, relationship with the health team and the patient, interest in the activities, and other criteria of the institution’s Medical Residency Commission (COREME)1),(3.
For the Brazilian Association of Pediatric Surgery (ABCP), the training in pediatric surgery aims to educate the specialist, previously trained as a general surgeon, to work as a pediatric surgeon in the following areas: neonatal, general, and urological pediatric surgery, pediatric oncologic, trauma, and video-assisted surgery. With this training, the pediatric surgeon will adequately advise families and peers in other specialties, define and perform clinical and surgical procedures, use technical resources, and perform appropriate interprofessional integration4.
With the institution of expected medical competencies, considering the DCNs, COREME and ABCP, tools and techniques can be developed to carry out the physician’s training, and the assessment of the acquisition of these competencies. The Milestone Projects of the North American Accreditation Council for Graduate Medical Education (ACGME) is a project developed with the American Society of Specialists for semiannual and/or annual monitoring of the developmental milestones of knowledge, skills and attitudes during specialist training5)-(8.
In this scenario, the objective of this research is to present an assessment instrument that can be used as a tool to evaluate the acquisition of competencies throughout the pediatric surgery residency in Brazil, following CNRM regulations and the Brazilian Association of Pediatric Surgery guidelines3),(4.
METHOD
This is a descriptive exploratory methodological study that combined the investigation of methods used to obtain, organize and analyze data, and discusses the development and validation of an instrument that is applicable, dependable, reproducible, reliable, relevant and appropriate from the technical and theoretical points of view.
RESULTS
This research was developed in three stages, designed according to AMEE Guide n. 8710. The AMEE Guides cover education topics related to medical and healthcare professions and provide information, practical advice and support. They are designed to be individually used by teachers to inform about their practice and are widely used in staff development programs.
Stage 1: Identification of the competencies required for the training of pediatric surgeons, according to the regulations of the Ministry of Education and the Brazilian Association of Pediatric Surgery. It is a survey of General Medical Competencies, Medical Surgery Competencies, and Medical Competencies in Pediatric Surgery through the analysis of international documents, such as CanMEDS Framework, Paediatric Surgical Education and Training Regulations, and the Milestone Projects3)-(5. The development of the assessment instrument is based on the Milestone Projects, with the stratification of competencies being carried out by residency training period, to be applied at the end of the first 6 months, at the end of the first and second years, and at the end of residency5.
Stage 2: Qualification of the instrument created using a semi-structured questionnaire, by a group of experts from the research institution, selected based on the following criteria: being a pediatric surgery specialist and a pediatric surgery residency service preceptor.
Stage 3: Content validation and quantification of expert acceptance. The assessment instrument was adapted to the Survey Monkey (R) program, as well as the Informed Consent Form, in the form of a semi-structured questionnaire, in which each acceptance item has a binomial answer (YES or NO), and space for considerations by the survey respondent. The invitation to participate in the research was sent by e-mail from the national pediatric surgery discussion groups: the Brazilian Pediatric Surgery Group (GBCIPE) and the Brazilian Pediatric Urology Group (UROPED), to quantify the instrument acceptance and evaluate dependability, reliability, applicability, reproducibility, relevance of the addressed topics, and adequacy from the technical and theoretical points of view.
Subsequent documental analysis of step 1, stage 1, called “Instrument Development”, comprising 13 sub-competencies to be developed during training were established4)-(6.
Medical Career Management:
Communication with the patient and family;
Communication with physicians and other health professionals;
Preparation for teamwork;
Informed consent;
Practice-based quality and safety improvement processes.
In stage 2, named “Instrument Qualitative Assessment”, of 7 selected experts, 4 effectively sent their considerations on the assessment instrument, totaling 44 considerations and suggestions for changes. At the end, the competencies were as follows:
Medical Expertise:
Bases of Pediatric Surgical Clinic;
General principles of pediatric surgery and preoperative preparation;
Hydration, analgesia, soft tissue infection, healing, judicious use of antimicrobials;
Venous access, gastrostomy, colostomy, surgical preparation of the newborn;
Complex drug management, management of postoperative complications, and co-management of critically-ill patients;
Complex case management and application of evidence-based medicine.
General and Neonatal Pediatric Surgery:
Pre- and postoperative care of appendicitis, hernias, hemangiomas, lymphangiomas, acute abdomen, and pediatric trauma;
Performance of the surgical procedures in item “a”, as well as pre- and postoperative care of: branchial arches, ovarian diseases, childhood abuse, intestinal invagination, Meckel’s diverticulum, cholelithiasis, and hypertrophic pyloric stenosis;
Performance of surgical procedures in item “b” and application of pre- and postoperative care of: congenital megacolon, gastroesophageal reflux disease, intestinal atresia, necrotizing enterocolitis, intestinal malrotation, neuroblastoma, gonadal tumors and teratoma;
Performance of the surgical procedures in item “c” and application of pre- and postoperative care, as well as surgical management of: esophageal atresia, anorectal anomaly, omphalocele, gastroschisis, biliary atresia, choledochal cyst, and abdominal tumors;
Video-assisted surgery in Pediatric Surgery;
Basic knowledge on video surgery equipment, and the clinical implications of video-assisted surgery
Assembly of the laparoscopic equipment, correct portal insertion, identification of clinical implications of laparoscopic surgery in the neonatal patient;
Performance of diagnostic procedures: laparoscopy, thoracoscopy, biopsies, appendectomy, identification of surgical complications;
Performance of complex procedures such as treatment of gastroesophageal reflux, cholecystectomy, correction of intestinal malrotation, treatment of surgical complications;
Innovative approaches in video-assisted surgery.
Pediatric Urology:
Diagnosis and surgical management: phimosis, acute scrotum, adhesions of labia minora (synechia vulvae), cystic scrotal lesions;
Pre- and postoperative care of: glans hypospadias, interlabial masses, hematuria, penile malformations, nocturnal urinary incontinence, nephrectomy, cryptorchidism;
Surgical procedures in item “b” and application of pre- and postoperative care of: vesicoureteral reflux, hydronephrosis, renal cystic disease, ureteropelvic junction stenosis, megaureter, renal duplication, posterior urethral valve, and heminephrectomy;
Procedures in item “c” and application of pre- and postoperative care of: penoscrotal hypospadias, epispadias, intersex, nocturnal enuresis, neurogenic bladder, bladder enlargement. Knowledge on the management of bladder and cloacal exstrophy;
Procedures in item “d” and management of kidney transplantation, neovagina, neopenis and exstrophies.
Chest surgery in pediatrics:
Diagnosis and management of laryngeal and tracheal obstruction, performance of adequate airway approach;
Pre- and postoperative management of: pneumothorax, pleural effusion, empyema, and tracheostomy.
Procedures in item “b”, and pre- and post-operative care of: pulmonary abscess, pulmonary necrosis, and pulmonary decortication;
Surgical procedure for the treatment of vascular rings and slings, bronchogenic cyst, bronchiectasis, mediastinal masses, congenital lobar emphysema, cystic adenomatoid malformation, pulmonary sequestration, and congenital diaphragmatic hernia;
Complex case management and application of evidence-based medicine.
Continuing Medical Education:
Continuing education and critical evaluation of the literature:
Critical evaluation of the literature;
Identification of resources available in the literature, identification of the limits of technical knowledge, use of a study plan;
Application of evidence-based medicine to the treatment plan;
Comparison of their results with medical literature;
Contribution to the literature regarding the management and treatment of complications of complex cases (only).
Professionalism:
Responsibility with the patient and Teamwork:
Punctuality, commitment, ethical principles;
Adequate recording in the patient’s file;
Compassion, empathy and respect for patients and families, colleagues, and other team members;
Application of standards in patient care, and reception of feedback on their attitudes;
Participation in institutional committees, and presence of ethical attitude;
Leadership, recognition of ethical implications, and management of complex issues.
Medical Career Management:
Communication with patient and family;
Listening ability and good communication in the clinical routine;
Checking Patient Understanding, provision of time for questions, and explanation of care plan;
Coping with stressful situations, bad news communication, and socio-cultural knowledge involved in different daily situations;
Knowledge on protocols for communication of bad news or death, knowledge on how to report medical error and use of a management plan for medical error management; educating patients and guardians regarding patient’s clinical problems;
Participating and assisting in solving challenging situations.
Communication with physicians and other healthcare professionals and teamwork:
Understanding the importance of teamwork;
Demonstrating knowledge of their roles in the patient care team;
Behaving effectively in a multiprofessional team;
Working with interdisciplinarity for better patient treatment outcome, responding appropriately to consultation requests;
Effectively guiding other professionals;
Informed consent:
Understanding the importance of the informed consent;
Involving patients and decision makers, getting informed consent for simple procedures;
Using appropriate language, and an interpreter if necessary, for shared decision-making, and obtaining consent for complex procedures;
Participating in multidisciplinary and conference teams;
Defining complex decision models.
In stage 3 of the research, called “Content Validation and Quantification of Expert Acceptance”, at the deadline for reception of the answers, the online questionnaire was evaluated by 40 pediatric surgeons, comprising a total of 2394 answers out of 50 competencies.
The total number of affirmative answers, with the respective percentage values in each evaluation item of the evaluation instrument is shown in Table 1.
Assessed Criterion | Overall Instrument assessment |
---|---|
Applicable | 379 (96.7%) |
Dependable | 296 (79.8%) |
Reproducible | 347 (93.3%) |
Reliable | 318 (85.5%) |
Relevant | 356 (96.0%) |
Appropriate from a theoretical point of view | 349 (93.3%) |
Appropriate from a technical point of view | 349 (93.6%) |
Overall | 2394 (91.2%) |
Source: Study Data, 2016.
After an overall analysis of the results, the instrument was considered applicable in 96.7% of the answers, reproducible in 93.3%, relevant in relation to the covered topics in 96%, technically adequate in 93.6%, and appropriate from a theoretical point of view in 93.3%.
DISCUSSION
To conclude stage 1, a survey of the overall and specific objectives of the residency programs in Pediatric Surgery was made, based on the definition of medical competencies, and following the regulations of the Ministry of Education, and according to the Brazilian Association of Pediatric Surgery4.
The construction was performed by similarity of meanings of the items, which allowed the synthesis and creation of an initial list with 4 competencies: Medical Expertise, Continuing Medical Education, Professionalism, and Medical Career Management. Subsequently, the minimum requirements for the training of the pediatric surgeon were analyzed, so that no essential items were excluded, and items that were repeated or complementary were not unnecessarily included, which would make the instrument too long, hindering adherence to the research.
The model instrument: “The Pediatric Surgery Milestone Project” provides a framework for competency milestones to be developed by the resident physician, to be evaluated every 6 months. It consists of 22 questions, distributed in progression levels from 1 to 5. For each period, the evaluation and reporting will imply the selection of the milestone that best represents the performance of each resident’s current attributes. Progressing from level 1 to level 5 means moving from beginner to expert. The selection of a level indicates that the resident can substantially perform the principles assignments of that level5. The training of the Brazilian pediatric surgeon includes several specific technical competencies, which are not covered by the American instrument; therefore, adaptation, rather than translation of the model instrument, was chosen. To better adapt the instrument to the Brazilian reality, it was suggested that the competency levels should be assessed at the end of the first 6 months of specialization, at the end of the first and second years, and at the end of the third year.
During the first year of training, two evaluations are expected: the first one taking place at the end of the sixth month, with a diagnostic function, because general surgeons from different training services may have developed different competencies during their initial training. The following evaluations take place annually and have a formative nature, because they follow the acquisition of competencies that are becoming increasingly complex, from the theoretical and technical points of view, as the specialization takes place9),(11. The interval between the evaluations is now annual, because the diseases that are part of the second and third-year resident’s assessment are less prevalent.
As some residencies offer an additional year of specialization, focused on the training of the ‘super specialist’, such as pediatric urology, pediatric liver transplantation, neonatal surgery, an additional column was created in the evaluation timeline, being related to the competencies expected only from fellows.
At the end of the first stage of the research, the instrument had 4 general competencies, subdivided into 13 specific sub-competencies, divided in 5 levels, evaluated at the end of the sixth month, at the end of the first, second, and third years, and an additional progression level, expected only from fellows, but which some third-year residents may reach. Thus, there was a total of 65 levels of competency assessment, and of these, 25 correspond to competencies that are specific of the specialty. Therefore, it is considered that the first 2 steps compared to those suggested by the AMEE Guide n. 87 were met at this stage5),(10.
In stage 2, an invitation was sent by email to 7 selected experts. Of the 6 who agreed to participate, 4 effectively sent their considerations on the assessment instrument within the recommended time frame for the study, totaling 44 considerations and suggestions for changes. Only 4 evaluators were accepted, because they had more than 80% agreement on the suggested changes.
At the end of the second stage of the research, it can be considered that steps 3, 4, 5 and 6 of AMEE Guide n. 87 for construction were followed, and the competencies were stratified into 4 overall competencies, subdivided into 10 sub-competencies. Each of them has 5 assessment levels, thus comprising 50 assessment items10.
In stage 3, the instrument was adapted to the Survey Monkey (R) program for online application. In this evaluation phase of the pilot project - last step of the AMEE guide n. 87 - the research was sent by email for acceptance of the research participation, along with the Informed Consent Form, by the Brazilian Group of Pediatric Surgeons and the Brazilian Group of Pediatric Urologists; specialists from all over Brazil had the opportunity to judge and evaluate whether the proposed items or sub-competencies were applicable, dependable, reproducible, reliable, relevant, technically and theoretically appropriate.
The results of step 3 suggests that the instrument can be applied in other pediatric surgery training services as a form of continuous and serial evaluation. Regarding reliability, the questionnaire had an 85% acceptance, suggesting that the respondents of the instrument agree that the items evaluated are appropriately approached. Regarding dependability (79.8%), the research instrument had a lower acceptance rate. When an item is considered dependable, it reflects the reality and can be considered as true. This can be translated by the fact that 79.8% of those who agree that the developed instrument represents the reality of the pediatric surgeon’s training, by training period. As the developed instrument was validated in a hospital with a long tradition in training pediatric surgeons, this can lead to the interpretation that the specialist training may not occur uniformly in Brazil.
It is worth mentioning that the present study contributed to the creation of the pediatric surgery competence, published em 202016.
CONCLUSIONS
Given the results of this research, an assessment instrument was developed, based on the Milestone Projects, following the objectives for training pediatric surgeons of the National Commission of Medical Residency and the Brazilian Association of Pediatric Surgery, for application in medical residency programs in Pediatric Surgery in Brazil.
The developed instrument was qualitatively validated regarding its content, with experts in the field, and the relevant changes were made, so that it can be continuously and serially applied to pediatric surgery resident physicians.
This instrument has been considered applicable, reproducible, relevant, and adequate from a technical and theoretical point of view. Regarding dependability and reliability, the instrument had a lower rate of positive acceptance in the overall assessment of the 4 major competencies.
Thus, it can be inferred that it is an important tool for formative evaluation and for determining it; it can be widely applied, helping to determine which training milestones have already been reached, guiding both residents and preceptors on the status of competence development during training.
Thus, it is a guiding tool for different pediatric surgery residency programs to promote effective evaluation and, consequently, to provide linear and homogenous competent training during the Pediatric Surgery Medical Residency programs in Brazil, as well as to guide the necessary adjustments to the services that offer residency in their specialty.