English (pdf)
Article in xml format
Automatic translation
Send this article by e-mail
Permalink
INTRODUCTION
Medical education has changed, especially regarding the student’s training method1. Due to the characteristics of the traditional training model, it is often impossible to experience more significant variability and complexity of clinical cases. In addition, another difficulty reported is the students’ insecurity when treating patients for the first time2. Thus, health education professionals have been looking for new techniques to improve students’ clinical skills and ensure the patient’s integrity.
Traditionally, in health education, simulations are carried out to train clinical procedures. For these simulations, instructors use mannequins, animals, and corpses. However, these practices have the disadvantage of requiring on-demand preparation and raise ethical issues3. In this sense, virtual simulators emerged as an alternative to traditional training. They allow the user to interact with a virtual environment that is similar to the interaction in the real world, reducing costs and enabling the experience of more significant variability of clinical cases.
There are minimum criteria for developing training with simulations, such as defining the expected objective, the target audience, the application usage scenario, the challenge difficulty, the main application subject, and the concepts related to it4. Thus, adequately designed simulation-based training can significantly reduce health professionals’ errors and improve patient safety2. In the literature, we identified studies about virtual simulators for different health fields that use technologies such as Virtual Reality (VR), Augmented Reality (AR), and Digital games, using tactile devices, specific googles/helmets and controls to provide immersion and interaction in realistic training of clinical procedures.
Therefore, this study aims to understand how health education uses virtual simulations for clinical training, focusing on the assessment of user skills. For this purpose, we carried out this integrative literature review. To the best of the authors’ knowledge, there have been no reviews of the scientific literature on the subject.
RESEARCH METHODOLOGY
The integrative literature review was chosen as the methodological procedure for the selection and analysis of research related to the subject. Each step will be detailed below, as well as the obtained results.
Research questions
Virtual simulators for training address skills to be developed and the several ways to assess their acquisition. Therefore, the following research questions (RQ) were defined aiming to contemplate the different scenarios on the topic: 1. How are the variables and parameters of virtual simulators defined?; 2. What are the assessed skills?; 3. How are these skills assessed?; and 4. How is the virtual simulator effectiveness verified for the acquisition of skills?
Papers screening
After defining the research questions, the next step was to select the studies for analysis. Terms commonly present in studies on virtual simulators for health training were used in the search to cover the largest number of articles and obtain a broad state-of-the-art view. As a result, the following search string was obtained: (“augmented reality” OR “virtual reality” OR “simulation” OR “simulator” OR “haptic” OR “haptics”) AND (“medical education” OR “medical training”).
The following digital libraries were searched: ACM Digital Library, IEEE Xplore, and PubMed, considering that the first two are important publication vehicles for Computer Science and Medical Informatics. The studies were selected based on the title/abstract search using the search string defined above. In addition to the cited databases, we also included articles that we deemed relevant or appeared in the selected studies’ references. Based on the search string and after reading the titles of the studies, we initially selected 210 articles. Finally, all articles were read in full and the exclusion criteria were applied. The defined exclusion criteria ware: 1. not addressing health procedures, 2. not addressing skill development for health professionals, and 3. dealing with non-digital simulations. Most of the excluded studies included non-digital simulations. In this last phase, a total of 56 studies were selected, which were relevant to our objective.
ANALYSIS AND DISCUSSION
Table 1 summarizes the main results obtained in the present study. The following sections answer the research questions.
RQ1: How are the variables and parameters of virtual simulators defined?
In this study, the variables of a clinical procedure mean the set of elements that constitute it, such as the instruments used and the professional’s performance. The parameters mean the values assigned to the variables, which can be the presence of a step or clinical instrument, as well as a range of acceptable values, such as needle depth and angulation. Virtual simulators usually use the variables and parameters to assess user performance, and they influence the accuracy and reliability of the simulator.
Most of the analyzed studies defined the variables and parameters of the virtual simulators with the help of experts in the area, which is equivalent to more than 70% of the selected studies. The participation of these experts took place through multidisciplinary research groups or by inviting them to participate in specific stages of the virtual simulator production. Their participation in the testing stage is also common. However, we observed that there is no defined standard regarding the participation of these professionals, and the publications do not provide clear indications about the methods and criteria for selecting experts. Furthermore, it is not clear how often their consulting activities takes place. Considering that the analysis and definition of a system’s requirements are cyclical, this step will require those involved in more than one moment, for example. In turn, the multidisciplinary teams theoretically allow the more active and frequent participation of professionals from different fields of activity, being able to identify variables and parameters with precision, influencing the reliability and accuracy of the final product. Concerning quantity, two studies that only consulted one specialist were identified. However, this should not be a limitation if there are more professionals involved in the validation step.
Another way that was identified to obtain the requirements is from the medical literature; around 13% of the studies did this. Only approximately 11% of the studies combined the literature with the participation of experts. The literature is one of the t resources traditionally present in professional training, so relying on its information is considered reliable, although the hands-on experience can add observations that will make the virtual simulation even more realistic.
The knowledge construction process consists of three stages: knowing, knowing how to do and knowing how to be. The awareness of education enables the learner to acquire information. The knowing stage operates in the sense of giving meaning to the theory, practically transforming it into knowledge, which is related to the stage of knowledge. Knowing how to do it is related to putting knowledge into use. That is, the development of skills related to the studied field. Finally, knowing how to be is about the learner’s attitude. It lies in deciding to put knowledge (knowing) and skills (knowing how to do) into motion. In this sense, the literature can assist in identifying variables, parameters, and clinical procedures steps. However, the granularity of the problem and the criteria for evaluating the identified variables and parameters are often not found in the books, making the participation of experts necessary to assist in modeling the parameters that will be inserted in the simulator and defining the assessment criteria for the variables. Only experts can assist in obtaining this information with a focus on knowing how to do and knowing how to be. We consider that the exclusive use of the literature can limit the establishment of parameters for assessment while relying exclusively on the expert can limit the reliability and accuracy of the simulation. The ideal situation is to consult both sources in a complementary way. We also consider important that teams involved in developing virtual simulations use validated techniques to define criteria for selecting experts, such as the Fehring model60.
We also identified a considerable amount of proposals with little or no detail on how they defined the virtual simulator variables and parameters. A total of 10 studies did not specify this information. This amount is higher than that of studies based on the literature. Although in half of these studies, the text suggests the literature, mainly related scientific articles, they do not specify how they extracted the information in question. The other half is about tools that share tactile force feedback, whose later tests cover aspects of performance or usability validated by students, and in one case, by experts. One of the studies focuses exclusively on describing the proposed technique without detailing the source of information.
Table 1 Summary of the main results
| Subject | Approach | References |
|---|---|---|
| Definition of variables | Consulting with experts | 5-44. |
| Consulting the medical literature | 3, 17, 19, 37, 39, 45, 46. | |
| Skills | Technical skills | 3, 5-14, 16-22, 24-59. |
| Non-technical skills | 6, 7, 12, 13, 15, 17, 34, 35, 38, 46, 49, 58. | |
| Hybrid skills | 6, 8, 13, 17, 23, 35. | |
| Assessment | Real-time alert/targeting mechanisms | 5, 6, 9-11, 13-17, 22, 24, 28, 29, 31, 33, 35, 42, 46, 49, 50, 53, 54. |
| User activity registration | 6-8, 10, 11, 13, 14, 17, 19, 27, 37, 38, 39, 56. | |
| Progress indicators | 11, 22, 24, 26, 43, 44, 51, 52, 53. |
Source: Prepared by the authors.
RQ2: What are the assessed skills?
Huang et al.61, in his literature review of VR-based virtual simulators for training in anesthesia, classifies the user skills practiced in simulators as technical skills, non-technical skills, and hybrid skills (Figure 1). Technical skills include psychomotor development, sequential knowledge of the procedural steps, and the characteristic instruments of the procedures, as well as time and safety during its practice. Non-technical skills are related to clinical decision-making, stress management, empathy with the patient, among other skills related to logic and professional conduct. Finally, hybrid skills address both technical knowledge and professional conduct.
Source: Huang et al.61 (p. 3).
Figure 1 Skills addressed in virtual simulators according to Huang et al.61
The selected studies showed a tremendous effort in the development of the learners’ technical skills, especially concerning procedural steps (45 of 56 studies), the handling of clinical instruments (38 of 56 studies), and dexterity (38 of 56 studies). In fact, virtual simulations emerge as an alternative to the traditional training model, in which the main focus is the development of technical skills. In this sense, although there is potential to address hybrid skills, this field is still little explored. Regarding speed and efficiency, although time is a determining factor in some clinical procedures, we observed that the studies prioritized the correct performance of the procedure rather than the completion time of the activity.
We identified three studies13),(17),(35 whose proposals consist of a patient’s virtual representation on a computer device. That is, these virtual simulations return to the user, in real-time, visual feedback of the symptoms of a simulated patient model, where decision-making and the speed and appropriateness of the diagnosis can cure or cause the death of this virtual patient. These studies, in addition to technical skills, work on the apprentice’s hybrid skills. In all, 11% of the articles address hybrid skills (6 of 56).
Regarding non-technical skills, the selected studies addressed clinical decision-making, diagnosis, teamwork, and stress management. The latter addressed the anesthesia procedure and seizures6),(13. Non-technical skills represented approximately 25% of the studies. It is reported in the literature that emotional factors, such as insecurity with the first patients, constitute significant challenges for the performance of new professionals62. In this sense, it is crucial that virtual simulators also help in the development of these skills.
We understand that the difficulty of having more constant participation of specialists is a factor that can influence the accuracy and development of new virtual simulators, especially the ones that address non-technical and hybrid skills, which require considering a greater number of variables. This concept was confirmed when it was observed that it is more common for studies whose team is a multidisciplinary one, which theoretically provides more significant interaction between professionals, to address the skills in question in their virtual simulations. For example, almost all selected studies that addressed hybrid skills were developed by multidisciplinary teams.
RQ3: How are the skills assessed?
In their study, Machado et al.63 highlighted the main challenges in building a virtual simulator for Medical Education, including the challenges related to skill assessment. Skill assessment allows assessing the users’ performance to identify whether they are prepared to perform the procedure in real-life situations. In order to return information about the users’ performance, it is necessary to analyze their actions during the virtual simulation. The capture of user actions can occur through usual devices such as the mouse and keyboard, and it is also possible to use more specific devices that acquire spatial position data. The simulators carry out the user’s skills assessment based on the collected data, either offline or online.
Offline skill assessment consists of recording the user’s actions in the virtual simulator for further analysis by an expert in the area, who generates a report and returns it to this user in a non-automatic way. On the other hand, online skill assessment monitors user actions by collecting data, such as angle and strength. It then compares them with performance classes previously defined by an expert in this procedure. This assessment method is integrated with the application, and the feedback occurs in real-time or after training64.
In their study, Burdea et al.65 presented the first proposal for the assessment of online skills in simulations, based on Boolean logic. This comprised a very simple assessment method that compared the diagnoses provided by users with the correct ones stored in the simulator. Later studies have proposed more complex assessment methods, which could treat the information granularity. The study by Machado et al.63, for example, presented a simulator for training in bone marrow collection based on VR, with 3D visualization, haptic interaction, collision detection, and user assessment. The authors performed a comparison between training assessment methods, in which the assessment tool based on the Fuzzy Bayes Rule provided high precision in the results. In the reviewed articles, we identified online skill assessment through: 1. real-time alerting/targeting mechanisms, 2. progress indicators, and 3. reports about the user performance in the virtual simulation.
The most often identified method of assessment in the analyzed studies is the real-time alert/targeting mechanisms. These mechanisms provide the user with some feedback, whether as messages, images, sounds, or even alerts to the instructor. After each user action, the simulator informs whether it is correct and how it is possible to improve it. The second most common form is the registration of user activities, which consists in generating a report at the end of the training, where it is possible to monitor successes and improvement points. Finally, there are the progress indicators, which provides: 1. scores for correct answers, 2. retention on a given task until completion, and 3. the possibility of repeating a particular task to improve practice. These indicators are objective ways of verifying whether the user achieved the expected performance. In addition, they act as motivating agents, as they adopt game-like characteristics.
There was a considerable number of studies (40% of them) whose objective is to present a new simulation technique, such as the texture and cutting of human tissues, or the advancement of an existing computational technique. These studies focus almost exclusively on engineering aspects, leaving the assessment of skills as a future task or approaching it superficially. Such studies do not detail assessment information. These virtual environments commonly bring only the tactile-force feedback. Although tactile feedback plays an essential role in aiding the development of clinical skills, in some cases, these studies do not take into account the online assessment process and condition the assessment of user skills to the presence of a supervisor (offline assessment). Approximately 43% of virtual simulators need instructors to carry out user skill assessments. We also observed that, regardless of this aspect, 56% of the analyzed simulators have their use conditioned to the instructor’s presence. Another point that was observed is that most studies use simulated data when performing tests37.
RQ4: How is the virtual simulator effectiveness verified for the acquisition of skills?
The verification of the occurrence of skills acquired by students is a point of attention in the studies. In simulators that use alert mechanisms, is assumed that the reduction of these messages indicates fewer errors, hence a behavioral change. However, we are aware of the importance of timely feedback. The same goes for progress indicators, as it is necessary to provide the student with a meaningful reflection about their performance to avoid resorting to trial and error. In performance reports, some studies do not clarify how the information is made available. In addition to being quantitative, this feedback must also be qualitative in the sense of presenting a clear textual (or visual) feedback; otherwise, the presence of an instructor will still be necessary to complete the training cycle.
Among the selected studies, there are 36 (64% of them) in which the presence of an instructor is necessary to provide some feedback on the user’s performance in virtual simulation. In 20 of these studies, it is assumed that an instructor is needed because, in addition to not providing feedback on student performance, these simulators do not have mechanisms capable of situating learners about the completion of their tasks. There were two of these studies10),(17 in which the instructor’s presence was only necessary for some simulation modules. In these cases, the presence of instructors was due to network interaction that allowed discussions between students and professionals.
Bloom’s taxonomy66 classifies learning as a plural and interactive phenomenon that co-occurs in the cognitive, affective, and psychomotor domains. Virtual simulations that operate on motor skills can directly contribute to achieving educational goals related to the psychomotor domain. The psychomotor domain deals with behaviors that imply the development of neuromuscular coordination. It related, therefore, to the acquisition of skills that combine muscle actions, cognition, skills to manipulate objects or perform a procedure67. Although Bloom and his team never defined a taxonomy for the psychomotor domain, others did it. Dave’s Dave’s classification68 for the psychomotor domain is the most often cited interpretation. It consists of five categories: imitation (observing the skill and trying to repeat), manipulation (following instructions, memorizing a procedure and being able to reproduce it), precision (performing the skill accurately and without help), articulation (combining skills to achieve a non-standard goal) and naturalization (unconscious domain of activity, when one becomes an expert).
The naturalization category is not typically covered in virtual simulations since to become an expert, the apprentice will need time of practical experience with actual patients. Regarding the precision category, as already mentioned, although computer simulations have the potential for self-guided use, the absence of this functionality is still common. It implies challenges in advancing the goals of the psychomotor domain, since the presence of an instructor is still necessary. Regarding the articulation category, although there efforts have been made in this direction, it constitutes a challenge for simulations to present greater variability in clinical cases, aiming to predict the wide range of possible clinical scenarios for the same procedure. This demand reinforces the need for constant and active participation of domain experts.
In addition to reducing costs and support the tactile and visual aspects, another differential of training mediated by virtual simulators is the possibility of using it at any time, without requiring supervision and receiving accurate and agile feedback about the user’s performance. In this sense, studies should further explore this potential. We understand that virtual simulators can be efficient tools for skill acquisition. However, feedback provided to the user about their actions during the simulation plays a fundamental role in the knowledge construction process. Therefore, these tools must provide personalized, relevant, and timely feedback on user performance. As noted, few studies have addressed the assessment of learning, and as it is an automatic assessment, few studies have implemented this functionality. Thus, we understand this potential area of the research landscape as a possible open problem.
Finally, when developing a virtual simulator for training, considering the user’s skills assessment, the results of this study show that it is essential to observe the following steps:
Design: a step that will deal with the survey and definition of the tool’s requirements. The criteria for selecting specialists must follow standards already reported in the literature, and the times and frequency of their participation must be specified. In addition, for the definition of variables and parameters, the knowledge established in the literature should be considered, complemented by the experts’ knowledge. We recommend developing documents such as concept and navigation maps and using case diagrams to help the interdisciplinary team communicate.
Skill feedback: a step that will verify the correct operation of the simulator, its educational effectiveness, and the user’s performance in the simulation. The presence of experts is necessary to validate the requirements defined in the design stage and collaborate modeling skill assessment methods by the simulator (online assessment). As for evaluating educational effectiveness, we observe that the single-use effects may not reflect the learning reality. Thus, it is essential to analyze how this construction of knowledge occurs over time and its effects on the work process. The acquisition of knowledge can be carried out in the short, medium, or long term, and what we observed in the studies are the single-use effects. Virtual simulators must provide personalized, relevant, and timely feedback to the student about their performance.
CONCLUSION
This review aimed to analyze studies about simulations in virtual graphic environments for training and education in the health area, focusing on the user skill assessments procedures. We conclude that using virtual simulators for training and assessment of clinical skills can be effective by using reliable and well-established variables and parameters, supporting teaching and learning in the field of health education. Considering the primary question of this research, we found that most simulator evaluations are not related to user skills but usability aspects. Thus, there is a significant research area to be explored in this direction, specifically related to user skill assessment, since this kind of evaluation has been little addressed in virtual simulators. It demands efforts to integrate research from computer science, engineering, and health areas.
