INTRODUCTION

The medical training process in Brazil and worldwide has been improving in recent decades, mainly due to changes in the structure of medical curricula. Medical Residency Programs (MRPs) in various specialties are included in this change process, which focuses on developing competency-based education for quality training, including the way competencies are taught and assessed¹. It emphasizes the outcome of the training process, prioritizing the health needs of the population and patient safety². Epstein and Hundert describe professional competency as “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served”³.

It is crucial that the essential competencies, specific to each specialty, are clear, well-structured, and known to all parties concerned, serving as the basis for the training process². A study conducted in Brazil for over two decades has highlighted the importance of establishing the competencies to be achieved by residents in each of the training areas of the Medical Residency Programs in Radiology and Diagnostic Imaging (MRP-RDI)⁴.

In 2017, the Brazilian College of Radiology (CBR, Colégio Brasileiro de Radiologia) published the Brazilian Protocol for Training in Radiology and Diagnostic Imaging⁵. More recently, the National Commission for Medical Residency (CNRM, Comissão Nacional de Residência Médica) approved and published the competency matrix for the MRP-RDI, establishing training objectives for each level of the specialty⁶. These documents recommend integrated training, covering all aspects of radiology, with the aim of standardizing training in the specialty.

Brazilian Radiology follows the specialty worldwide; however, it has its own characteristics and has been standing out in the teaching and research areas. A peculiarity of Brazilian Radiology is the practice of Ultrasonography (US), one of the imaging methods that has achieved great development in the country⁷. In this regard, a recent study involving several MRP-RDI in the United States affirms that, despite attempts to improve the Ultrasonography curriculum for radiologists, standardized training in this subspecialty is still lacking in that country⁸.

Similarly to the Accreditation Council for Graduate Medical Education (ACGME)⁹, the European Society of Radiology (ESR)¹⁰ also organizes the curriculum for training programs by medical area, encompassing all subspecialties. The ESR recently published an article containing recommendations for best practice in the use of Ultrasonography in Europe¹¹. It advocates that radiologists should have a leading role in the use of the method because they deal with medical images full-time, and that there is a need to standardize the practice for both radiologists and non-radiologists. Previously, another study¹² had already demonstrated the need to increase the quality and visibility of Ultrasonography performed by radiologists in Europe, getting ready to be at the forefront of new applications of this imaging technique.

The present study aims to establish the essential items of a competency-based curriculum, specialized in Ultrasonography for the training of radiologists, through the creation of a competency matrix. The prospect is to improve the training process of MRP-RDI residents at the educational institution where the research took place, during the internship in the Ultrasonography sector, by determining the competencies to be achieved, aligned with the teaching-learning strategies and evaluation system necessary for their development during training.

METHOD

This study has an exploratory and descriptive design and was carried out in two stages. In the first stage, a documentary analysis was carried out on guidelines and protocols established for the MRP-RDI, aiming to collect essential topics and contents that could comprise the competency matrix in Ultrasonography for a MRP-RDI, in addition to the teaching-learning and assessment strategies aligned with the proposed topics. The research of documents was carried out in official sources available on the internet with free access, mainly on the websites of CBR⁵, ACGME⁹ and the ESR¹⁰, also including the CNRM⁶ guidelines for the MRP-RDI.

In the designation of the competencies, we sought to use Bloom’s Taxonomy^16),(17 to identify the level of the cognitive domain to which they belonged. For example, one of the objectives described in the initial matrix as a general competency determines that the student, at the end of the course, must “know and adequately handle the available Ultrasonography equipment, demonstrating knowledge of its functions and ensuring image quality”. There is a progression in the level of complexity with a single objective: knowing  handling  demonstrating  ensuring. This version was presented by the researcher to the other members of the research team and to the MRP-RDI supervisor, all of them with expertise in health education, as a pilot to obtain consensus, serving as a pre-test. Thus, the initial version of the competency matrix to reach a consensus was modified to improve its clarity, structure and explanations for the consensus.

In the second stage, the competency matrix in the area of ​​Ultrasonography for a MRP-RDI was structured using the modified Delphi method, a research technique that seeks to converge opinions among a group of experts on a given topic not widely explored in the literature. This type of collective judgment is very sensitive to the participants’ motivation and their knowledge of the subject to be addressed^13),(14. It uses an interactive questionnaire that requests anonymous quantitative responses supported by justifications and subjective information. The quantitative responses receive simple statistical treatment and are returned to the participants in a new round of questionnaires, together with comments on open-ended questions. This feedback allows the exchange of information among the participants, leading to a convergence of opinions and the subsequent construction of a collective response.¹³ The exchange of information, the anonymity of responses and the possibility of reviewing individual views based on the group’s viewpoint are essential characteristics of the method.¹⁴ The methodological sequence of the present study is outlined in Figure 1.

Source: the authors

Figure 1 Methodological Scheme 

The inclusion criterion for the panel of experts included being a preceptor of the MRP-RDI of the educational institution where the research was conducted, specifically in the area of ​​Ultrasonography care. Preceptors who were absent from their duties during the study period for any reason were excluded. Nine preceptors were contacted individually, corresponding to 100% of those who met the inclusion criteria, and, after explaining the objective and method used in the research, all agreed to participate by signing the Free and Informed Consent Form (TCLE).

The first two rounds of questionnaires were conducted digitally using a form created using the Google Forms® website, of free access, the link to which was sent to participants by email. The first questionnaire was divided into two parts: the educational profile and length of experience of the preceptors participating in the research and the evaluation of the components of the initial semi-structured matrix. The evaluation of the items of the initial matrix in the first round was carried out using a five-point Likert scale, ranging from 1 (fully disagree) to 5 (fully agree), having the score of 3 as neutral, with space for comments in open-ended questions immediately after each group of questions. Unlike the responses limited by the Likert scale, the comments were not mandatory.

In each round, the degree of agreement was defined, considering the responses in the “agree” or “fully agree” strata. In the first two rounds, consensus was considered for each item analyzed when the agreement rate was at least 90%. Thus, the matrix was presented again for the next round, when the panelists could express new perceptions about the items that did not reach a consensus. In the subsequent rounds, items that, even with a total absence of disagreement, had suggestions for changes according to the open-ended questions, were also considered.

In the 2^nd round, the evaluation of the re-presented items was also carried out using a five-point Likert scale, with space for comments in open-ended questions. However, unlike the 1^st round, in addition to the neutral opinion on the Likert scale (3 - neither agree nor disagree), in the category of competencies by medical area, another option was added: “I prefer not to give an opinion because I have no affinity with the specific medical area”. The participants were told that by choosing this option, they would be expressing a neutral opinion, but without any disagreement, based on a lower affinity with the item proposal.

The 3^rd round was divided into two phases. In the first phase, the suggestions presented in the 2^nd round that could contribute to structuring the matrix in the form of a workshop were analyzed, generating a new questionnaire. Structuring the initial competency matrix was the objective of the second stage of the methodology, and the suggestions given by the participants should then be analyzed by all participants before creating the structured matrix. This questionnaire contained only the items in the initial matrix that had not yet reached a consensus and those that were the targets of the suggested changes. In the second phase, the new questionnaire was physically delivered to the participants, taking advantage of the in-person workshop environment, facilitated by the reduction of the questionnaire in this round, while maintaining anonymity. There was room to express an opinion between two points: “I disagree in whole or in part” or “I agree in whole or in part” and it was advised that the new threshold for consensus would be greater than 50% of agreement, considering the refinement obtained throughout the rounds. Upon reaching a level of convergence among the panelists after the third round, the structured Competency Matrix was sent by email to the participants, together with the summary of the research process.

The study was approved by the Research Ethics Committee (CEP-HUOL) under Certificate of Presentation for Ethical Appreciation (CAAE) number 54181421.6.0000.5292.

RESULTS

The initial version of the matrix, containing 150 items, included the recommendations of the CNRM⁶, CBR⁵, ACGME⁹ and the ESR¹⁰ guidelines for MRP-RDI, in addition to the protocol for Ultrasonography courses of the CBR¹⁵. The items in the matrix were distributed into three main categories:

To create the questionnaire, each category of competencies in the initial matrix was subdivided into groups of questions, as shown in Table 1.

The three rounds of questionnaires took place between November 2022 and April 2023. Participation was 100% in the first two rounds and 89.9% in the third one. The participants’ training profile and length of experience are specified in Table 2.

Of the 150 items analyzed in the 1^st round, only 43 (28.7%) did not achieve consensus and/or were the subject of comments. The highest agreement rate was found between general competencies and their teaching methods, medical areas with the greatest affinity among the specialists, and competencies for the first year of the medical residency. The highest disagreement rate was found between assessment methods (frequency of written exams 4x/year and resident participation in scientific research) and between items with little or no opportunity for practice, such as safely differentiating fibroids from adenomyosis, evaluating ultrasonography markers for chromosomal abnormalities in the exam between 11 and 13 weeks and 6 days of gestation, four items related to penile evaluation, and organizing and conducting ultrasonography services at the end of the third year.

Some comments from participants in the open-ended questions directly reflect the highest disagreement rates, such as, for example, in relation to participation in scientific research as a teaching or assessment method. It was stated that “scientific research should not be considered a mandatory teaching or assessment method, but should only be encouraged, with the development of critical thinking as a fundamental item.”

The results of the 1^st round demonstrated the first impression the participants had about the content of the initial competency matrix. A reflection was still necessary, based on the group’s responses. Thus, the 43 items that did not reach consensus in this round and/or were the subject of suggestions were grouped together to constitute the questionnaire for the 2^nd round. Table 3 shows the items with the highest disagreement rate in the 2^nd round.

There was a greater contribution to open-ended questions in the 2^nd round, when eight participants made 33 comments, while only four participants made 18 comments in the 1^st round. The new comments corroborated the initial perceptions. Based on this analysis, the structuring phase of the competency matrix proceeded to the 3^rd round. Changes were proposed in nine items (Chart 1), eight regarding the wording and one regarding the level of complexity.

“To participate in research activities” as a general competency remained an essential item, but its wording was changed to “To demonstrate interest in participating in research activities”. It was considered that it would be very important, and even a condition, for preceptors to encourage the development of this competency. The participants’ suggestions were taken into account to avoid using terms in the wording that indicated mastery of the practice, such as “to obtain excellent images” and “accurate measurements”, “to safely diagnose”, etc. “To recognize perirenal and adrenal anomalies” was changed from basic to advanced level in the assessment of the urinary system. The assessment of specific lesions in Mastology and the Musculoskeletal system required clarification regarding the limitation of the Ultrasonography method in relation to other imaging methods.

After the 3^rd round, 96.6% of the items in the initial competency matrix analyzed achieved consensus, with a high agreement rate (75 to 100%). Only the items described below did not reach the consensus determined for this round (disagreement rate greater than or equal to 50%) and were eliminated from the initial competency matrix:

DISCUSSION

The construction of a specific Competency Matrix for Ultrasonography, as carried out in this study, is unprecedented in the national scenario and aims to assist in the training of MRP-RDI resident physicians for the challenges of specialized medicine in such a broad area. According to a recent European study, there are discrepancies in the use of this imaging method and a need for standardization of practice¹¹. It is true to say that this construction becomes relevant when Ultrasonography can be performed by both radiologists and non-radiologists, as observed in the profile of the research participants, reflecting what is practiced in Brazil and worldwide.

The guidelines established by the competency matrix prepared by the CNRM for MRP-RDI⁶, encompassing all imaging methods and without a specific and detailed approach to Ultrasonography, may be insufficient for the training of specialists in this area, mainly because they do not take into account the peculiarities of this imaging method^8),(11 and its scope in the national scenario⁷.

Based on the assessment of essential competencies necessary for professional development in the studied area, some topics need to be discussed, mainly those that have been the subject of divergent judgments. A recent analysis of systematic reviews on the Delphi method cites possible quality criteria for the methodology that can be observed for this study. The stability of the research participants’ responses and the reflections on how to manage divergent judgments are some of the mentioned criteria that were analyzed in this discussion¹⁸.

As for the methods used to evaluate resident physicians, some participants did not agree with the performance of quarterly theoretical tests and participation in scientific research did not seem essential to them as an evaluation process. This last aspect is probably due to the lack of a common practice of scientific research in the field of practice. To standardize these issues, a recent Resolution of the CNRM¹⁹ determines the evaluation criteria for resident physicians, among them, periodic performance evaluations every four months, covering the three domains of evaluation (knowledge, skills and attitudes); scientific research would be considered a possible evaluation method, at the discretion of the Medical Residency Committee (COREME, Comissão de Residência Médica). In this context, this study is contemporary and aligned with the regulations, including the recently published one.

Although the encouragement of scientific research within medical residency has been increasing in Brazil and worldwide^7),(20, some difficulties have been reported. Studies emphasize among these difficulties the low priority in relation to the demand for assistance, the lack of protected time for interested residents, as well as poorly prepared advisors^20)-(22.

In practice, the area of ​​research is mentioned as a gap in several MRP-RDI⁴, but the search for solutions should encourage a transformation of this context²³. Thus, the item “participate in research activities” as an essential item among the general competencies was maintained, only changing the wording to “demonstrate interest in participating in research activities...”. This corroborates the profile designed for “the ideal resident in radiology and diagnostic imaging”²⁴, who takes advantage of all opportunities offered to research. Therefore, the authors understand the perception of the preceptors participating in the study on this item, since the activities developed with the residents are essentially focused on in-service training, according to most of the MRPs.

Another topic that can be emphasized, since it was the target of several comments, is related to the training of residents in what they consider essential. While participants agree that teaching and assessment methods or some content should be reinforced in practice, they also consider the use of terms that require mastery of competency to be inappropriate. Terms such as “precise measurements”, “excellent images”, “diagnose safely” should be avoided because “these are sometimes difficult competencies, even for more experienced specialists”, and “resident training does not prioritize competencies in rare and difficult diagnoses”. These opinions are supported by the first concepts of competency-based education in its multidimensional aspect^1),(3, when the search for the implementation of this type of quality training emphasizes the result of the application of knowledge in the teaching-learning process²⁵. Among radiologists, competency must be directed towards a more assertive diagnosis, focused on the skills and knowledge necessary for clinical reasoning and decision-making².

A recent study corroborates this topic by seeing the need for updated approaches that give meaning to the development of the residents’ activities. It directs the teaching-learning process to the students’ profile, aiming at better performance²⁶. This aspect can be even further emphasized when talking about emerging topics, especially if they bring concerns and require adaptations, such as, for example, a pandemic or the use of artificial intelligence in some medical areas^27)-(29.

A participant’s comment raised an issue that may justify the way in which the competency matrix was structured at the end of the research. He considered the difficulty of having access to equipment with updated technology and the need to train preceptors in specific areas, sharing the problem with the hospital management area. It was said that “some competencies, to be considered essential, require adequate equipment and a preceptor with expertise in the subject”. Some items, when eliminated from the initial competency matrix, should stimulate the reflection on the situation in the practice scenario.

Guidelines established in some Brazilian University Hospitals for the exercise of preceptorship reinforce the need for incentives for training and updating preceptors in new pedagogical approaches³⁰. Likewise, studies have shown that characteristics of the environment and actors can influence the curriculum, which should be customized to reflect local expertise with its resources and learning opportunities^24),(31),(32. Among the participants in the study, it was found that only 22.2% had undergone training for teaching, characterized by the completion of a Master’s Degree. The lack of pedagogical training among the preceptors represents a major challenge in our country, as described in the literature³³. To meet the need for training, including the development of teachers and preceptors for teaching activities, Higher Education Institutions have developed strategies that facilitate such actions³⁰. In this context, stricto sensu postgraduate courses in health education and specialization courses in preceptorship have acted as facilitators of this process. Thus, it is worth noting that in the hospital where the research was developed, most of the preceptors had already participated in training workshops to introduce a new assessment culture encompassing knowledge, skills and attitudes. This moment represented a milestone for the teaching-learning environment of the MRP-RDI, when an assessment instrument of a formative nature was created to be applied in all areas of radiology, including Ultrasonography³⁴. Similarly, in a process of implementing a new competency-based training program for a medical residency in Anesthesiology in Canada, it was established that, among the various objectives to be followed, there should be a faculty development program to support the changes²⁵. Since the preceptor is a health professional-educator, they have a dual and complex commitment to care for the health of the population and, at the same time, to care for the professional training of undergraduate and graduate students in the health areas who care for others. Here, it is worth noting that training programs that raise the level of preceptorship activities in healthcare institutions should be continued.

The process of building a competency matrix based on customization based on the practice scenario can be better explained by considering that some MRP-RDI rotations are particularly more difficult. As an example, an American study shows that few residents in the area were interested in Interventional Radiology. Thus, by encouraging practice in this field during medical residency, requiring more training, there was a greater demand for the subspecialty³⁶. Pedagogical approaches with clear and well-structured objectives also contribute to improving the residents’ performance in underdeveloped topics. This was clearly demonstrated in an American study that found an improvement in the level of students in breast imaging sections when a curriculum based on “milestones” of progressive competencies for learning in the area was established³⁷.

These examples demonstrate that some activities that are not practiced in a given environment, or deserve to be better developed, can be the focus of further studies on how to include them among the essential competencies. However, both the determination of the items considered essential for a Competency Matrix and the elimination of items that did not achieve consensus at the end of the research can contribute to reflection. Although the competency matrix constructed ended up being comprehensive, some important issues may have been neglected in the dialogues between the participants. This study has as a limitation the fact that all participants belong to a single center. However, its results are relevant as an initial step that will allow the discussion and adaptation of the US competency matrix to other training centers. The incorporation of other cultures and practices, such as those related to teaching and technical expertise, can generate new ideas. It should also be considered that the participants, even though they are experts working in the field of Ultrasonography, represent only one source of information, limited to the context of preceptorship at the institution under study. Other results can be achieved by including other sources of information, such as opinions from professors and residents, the latter as end users of the teaching-learning process³⁸. Engaging residents in the pedagogical process can contribute to a better understanding of the competencies for their training and promote sustainable changes in the curriculum^27),(39. This proposal can surely be a stimulus for other actors and similar teaching centers to join in a future, more extensive study.

A relevant concern in the construction of this competency matrix was the inclusion not only of programmatic content, but also of teaching and assessment strategies that allow the engagement and active learning of resident physicians²⁷. The application of a competency matrix with this configuration should determine the alignment of objectives, instruction and assessment to attain the resident’s congruent behavior and boost the teaching-learning process. Although some of these strategies were well discussed by the participants in this study, more efforts should be made to strengthen those already used and listed in the Ultrasonography Competency Matrix. The design of each strategy needs to be contextualized, contributing to the objectives of the program, striving to achieve the desired quality and satisfaction of all actors³⁸.

CONCLUSION

The Competency Matrix in the area of Ultrasonography for a MRP-RDI was structured to represent a relevant instrument for competency-based training and improvement of the quality of professional practice based on national and international guidelines that were analyzed and adapted to the regional reality. Furthermore, it can be validated by other MRP-RDI in the national scenario, contributing to the development of a specific curriculum and improvement of training focused on Ultrasonography.