INTRODUCTION
Communication and Information Technologies (ICTs) started being used to help the acquisition of knowledge and facilitate the teaching-learning process1. The use of technology through mobile devices or mobile learning (m-learning) is known to offer many benefits, such as access to content anywhere and at any time, the ability to perform tasks and activities and note down ideas or queries on the Internet. Teaching through m-learning enables the simulation of complex systems, along with the benefit of promoting a large number of perspectives with a high quality of visualization and interaction2. In this context, ICTs, according to each modality, can enable the creation of new learning environments, allowing remote and/or online access to training platforms3, increasing communication and distance learning. Thus, these support tools incorporated into an electronic health management system exponentially increase the growth of medical information acquisition4.
There are several practical training methods for orthopedic surgeons, including hands-on training with synthetic bones or cadavers, and the use of software and computer simulators for planning and simulating situations in 3D environments5. This reduces financial costs and brings about improvements in skills development, the handling of surgical instruments, competence to reach the learning curve earlier, better assessment of the acquired techniques, and perhaps most importantly, the maximization of the patient’s safety6.
Within the ICT-based learning modalities, cognitive simulation is one of the most recent examples of medical training innovation. It is the process by which medical residents or students evaluate and rehearse actions within their minds without physical movement and, among them, is the training related to solving questions for theoretical-practical training. It is hypothesized that residents may improve their intraoperative performance and surgical skills through appropriate preoperative cognitive simulation, with or without appropriate preceptors to help them7.
The use of smartphone applications for the orthopedic surgeon training is a powerful tool for improving the quality of these professionals’ training8. Currently, 67.1% of medical students choose an application to help them with their daily academic doubts, and 42.2% use it to assist them in their professional practice9.
Among the several approaches to the teaching-learning process using mobile applications, the knowledge acquired through the resolution of various issues, also called retrieval learning, can be highlighted, which has proved to be more effective than just reading the content to be studied10. A search was conducted in the virtual stores Applestore® and GooglePlay® and it was found that most of the applications available for the orthopedic student based on retrieval learning were not properly scientifically validated. Therefore, the aim of this research was to develop and validate an application for the improvement and training of orthopedics and traumatology residents, which allows the resolution of preparatory questions for the Brazilian Board of Orthopedics and Traumatology (TEOT) exam.
MATERIALS AND METHODS
App development
The development of this application involved the following steps: review of the available education and training applications in orthopedics and comparison with evidence-based practices; creation of an initial database of questions based on the bibliography proposed by the Teaching and Training Committee (TTC) of the Brazilian Society of Orthopedics and Traumatology (SBOT); selection of strategies to change behavior and guidelines to be used in the application; development of the interfaces and designs of the application; and finally, selection of participants for the application of the validation test.
The application was developed in both Android® and iOS® platforms. Regarding the development, the following tools were used: IDE® (Integrated Development Environment) Android Studio; Android® Software Development Kit (SDK); the Android® Emulator System with Google's Application Programming Interface (APIs); and the OpenCV (Open Source Computer Vision Library) library for the development of the image processing functions present in the application. The application allows the customization of the content of the simulated questions, the visualization of the elapsed time and right or wrong answers with the respective bibliographic references. Also, the user’s results progression can be followed.
The participants were randomly selected using Microsoft Excel® software. They were then divided into three groups: group I, 4-year medical students of the Christus University Center (Unichristus) (33 participants) and the Federal University of Ceará (UFC) (33 participants); group II, residents of orthopedics and traumatology from the state of Ceará (33 participants); group III, consisting of orthopedists and traumatologists who are members of SBOT and medical residency staff (33 participants). This sample size was calculated based on the study conducted in Leipzig, Germany9, which showed the preference of applications for health training, when evaluating two groups of students, divided among those who want to use an application (69.9%) versus those who do not (28.0%); it was considered necessary to evaluate a sample of 33 individuals per study group in order to obtain a sample that represented 90% of power and 95% confidence of the null hypothesis in this study (Fleiss method with continuity correction). Calculations were performed using the Open Epi Stat Calc® (http://www.openepi.com/Menu/OE_Menu.htm).
The inclusion criteria were: age above 18 years old and participants who signed the Free and Informed Consent Form. Those participants who wished to discontinue their participation in the study were withdrawn from the study.
Regarding the ethical aspects, the basic principles of ethics in human research were considered, such as autonomy, justice, beneficence, and nonmaleficence. The study was submitted to the Human Research Ethics Committee (CEP) of Unichristus, protocol number 65487617.5.0000.5049. There are no conflicts of interest to be disclosed by the authors.
Data Collection
The questionnaire (Appendix A) was organized into three sessions: 1) related to the user’s authorization and experience with applications; 2) related to the SUS® (System Usability Scale) questionnaire11, validated in Portuguese to quantify the applicability of the developed application12; and 3) related to the importance of the application in learning, being developed based on Davis Technology Acceptance Model13 and the Likert Scale, which measures the system practicability level perceived by users during the resolution of questions and evaluates the usefulness of specific questions about orthopedics and traumatology, as well as learning.
The data were tabulated in Microsoft Excel® and exported to the Statistical Package for Social Sciences (SPSS) software, version 17.0 for Windows®, where the analyses were performed with a confidence interval of 95%. The mode, absolute frequency and percentage of the variables studied were cross-checked using Fisher's exact test or Pearson's chi-squared test.
RESULTS
The developed application was named Quiz Ortopedia ®, and its patent was registered in the National Institute of the Industrial Property (INPI) for iOS® (registration number BR 512018000006-1) and Android® (registration number BR 512018000005-3).
The application allows the user’s registration with name, e-mail, year of residence, a 6-digit password and password confirmation.
On the initial screen, it is possible to choose between a customized test, a simulation according to the suggestion by SBOT, a summary of the user's profile and information about the development and version of the application.
Regarding personalized questions, the user has access to main topics (Orthopedics, Trauma, and Basic) and subtopics divided mainly by body joints (hip, knee, etc.). It is also possible to choose several simultaneous topics and then one can choose the number of questions and the timing (per question or total time). Throughout the resolution of the questions, the user will be able to see the elapsed time, the total number of questions, as well as the number of correct or incorrect questions up to that time. After that, a result in percentage should be displayed and the user is given the option to access the comments about both right and wrong questions. In the user’s profile, there is a screen with the results, so that the user can check study progression up to that point.
The images of some representative screens of the application are depicted in Figure 1.
Regarding the profile of the study participants and their previous experiences with the use of applications, 55.3% of the participants (73/132) have iOS® as their operating system, while 44.7% (59/132) use Android®. All 132 participants reported that they had already used some type of application on their smartphones, and only 6 (4.5%) mentioned never having used applications for academic purposes (Table 1).
Occupation | ||||
---|---|---|---|---|
Student | Resident | Orthopedist | p-Value | |
Operating System | ||||
Android | 32 (48.5%) | 18 (54.5%) | 9 (27.3%) | 0.057 |
IOs | 34 (51.5%) | 15 (45.5%) | 24 (72.7%) | |
Used application for academic purposes | ||||
No | 3 (4.5%) | 0 (.0%) | 3 (9.1%) | 0.208 |
Yes | 63 (95.5%) | 33 (100.0%) | 30 (90.9%) |
* p <0.05, chi-square. Data expressed as absolute and percentage frequencies.
In group I, 63 participants (95.5%) had already used mobile applications for academic purposes. In group II, all of them had already used the mobile phone with the same purpose, and in group I, 30 (90.9%) used academic applications (Table 1). The type of operating system (iOS® or Android®) showed a significant statistical difference between the groups (p-value <0.001). The use of applications for academic purposes did not show a statistical difference between the groups.
When we analyzed the usability of the application, evaluated according to the SUS® questionnaire, we observed that it showed an average SUS score of 84.2 (standard deviation of 10.8) with a margin of error of 1.9. To verify the reliability of the obtained results, Cronbach's alpha coefficient was used14. The value obtained for this coefficient was 0.797, showing the sample had a good level of reliability.
The analysis of the answer to each question in the SUS questionnaire (part II) is found in Appendix B. We observed a statistical difference between groups regarding the following statements: 1) I think I would like to use this system frequently, in which the resident group entirely agreed with this statement, while the other groups only agreed with the statement (p-value = 0.018); 2) I found that the various functions in this system were well integrated, in which the resident and the orthopedic groups entirely agreed with this statement, whereas the academic group only agreed (p-value = 0.024); and 3) I was very comfortable using this system, in which the resident and the orthopedic groups fully agreed with this statement, whereas the academic group only agreed (p-value = 0.027).
The analysis of part III, regarding the importance of the application in learning, evidenced differences between the groups with statistical significance for all questions (0.001-0.032), except for question 6 (p-value = 0.177) (Table 2).
Occupation | ||||
---|---|---|---|---|
Student | Resident | Orthopedist | p-Value | |
1) Is problem-solving training part of learning? | ||||
Neutral | 1 (1.5%) | 0 (.0%) | 0 (.0%) | 0.003 |
Agree | 26* (39.4%) | 3 (9.1%) | 4 (12.1%) | |
Strongly agree | 39 (59.1%) | 30* (90.9%) | 29* (87.9%) | |
2) Can the resolution of questions help in the preparation for the Board of Orthopedics and Traumatology (BOT) exam? | ||||
Neutral | 7* (10.6%) | 0 (.0%) | 0 (.0%) | 0.001 |
Agree | 23* (34.8%) | 3 (9.1%) | 5 (15.2%) | |
Strongly agree | 36 (54.5%) | 30* (90.9%) | 28* (84.8%) | |
3) Do I believe that the practicality of a mobile application on problem-solving in orthopedics and traumatology could help in preparing residents for exams? | ||||
Neutral | 1 (1.5%) | 0 (.0%) | 0 (.0%) | 0.005 |
Agree | 25* (37.9%) | 3 (9.1%) | 4 (12.1%) | |
Strongly agree | 40 (60.6%) | 30* (90.9%) | 29* (87.9%) | |
4) Does the application seem to be a useful technology in the theoretical development of the orthopedic resident during their training? | ||||
Agree | 26* (39.4%) | 6 (18.2%) | 6 (18.2%) | 0.027 |
Strongly agree | 40 (60.6%) | 27* (81.8%) | 27* (81.8%) | |
5) Can the application be used as an auxiliary method of learning for medical students studying Orthopedics and Traumatology? | ||||
Disagree | 2 (3.0%) | 4 (12.1%) | 7 (21.2%) | 0.032 |
Neutral | 1 (1.5%) | 3* (9.1%) | 2 (6.1%) | |
Agree | 32 (48.5%) | 17* (51.5%) | 14 (42.4%) | |
Strongly agree | 31* (47.0%) | 9 (27.3%) | 10* (30.3%) | |
6) Can the application be useful as an auxiliary learning method for general orthopedic physicians? | ||||
Disagree | 1 (1.5%) | 0 (.0%) | 3 (9.1%) | 0.177 |
Neutral | 2 (3.0%) | 2 (6.1%) | 0 (.0%) | |
Agree | 34 (51.5%) | 15 (45.5%) | 19 (57.6%) | |
Strongly agree | 29 (43.9%) | 16 (48.5%) | 11 (33.3%) |
* p <0.05, chi-square. Data expressed as absolute and percentage frequencies.
DISCUSSION
A substantial increase in the availability and use of mobile applications for smartphones and tablets for medical education has been reported in recent years, including the handling of work tools and care in the learning mode15. They are of great importance for enabling remote and online access to the information, allowing greater flexibility in learning. Students are more comfortable regarding when, how and where to study, making it possible for them to escape the traditional method of teaching conferences in the classrooms16. This teaching alternative can bring safety to the patient treated by the future doctors, since it challenges students regarding decision-making4.
The advantages for the medical sciences include the encouragement of and engagement in independent and group learning, as well as flexibility in the method of study17. The use of ICTs in the medical area also includes improved patient care and multi-professional communication18.
Among the disadvantages that can be mentioned, in addition to the cost of developing the technology, are the failures of the mobile device method, such as access to multiple contents, offering distraction through other tools, or even superficial learning, inadequate screen size and insufficient memory, the lack of validation and, sometimes, lack of scientific evidence related to some application categories7. However, with the improvements in science, it is believed that many of these difficulties can be solved19.
In recent years, the growth of ICTs and their involvement in medical education has become significant, revolutionizing the access to information and learning methods. In previous literature review studies, it was observed that smartphone application technology can be integrated into successful orthopedic practice18. However, currently, the market does not have a large availability of applications targeting this specialty. Among users of these technology devices, approximately 50% use the Android® 22.5% use Blackberry® and 18.5% use iOS / Apple®(20 operating system. Among orthopedic surgeons, 84% use smartphones, mostly iPhone® (55%) and 53% use it in clinical practice. At the time of the survey, there were 61 orthopedic applications for the iOS / iPhone® platform and 13 applications for the Android® platform21.
Moreover, there are concerns about the content available in this tool, since published information validation tools are pointed out as a challenge to be put into practice21. In this perspective, a study evaluated the main applications used by orthopedic specialists in hand surgery. Among the educational purposes, the reading of scientific journals and recent publications were the main modes of teaching and medical training. As a result, there was no application for direct resident training regarding the resolution of issues or clinical situations22.
As for other studies, it could be inferred that, concerning orthopedic applications for teaching and medical training, the great majority consisted of guidelines, training of surgical techniques and videos3),(20, as well as for anatomy, medical calculators, laboratory results and drugs and medicines23. A pilot study of a simulation application aimed at the management of septic arthritis for orthopedic residents showed positive results when comparing the percentage of correct answers in the tests versus the correctness of the simulations, concluding that the subject training application has great potential to educate and evaluate the user in an efficient and cost-effective way24.
Faced with this brief bibliographic review, the Quiz Ortopedia ® is a revolutionary application for teaching and medical training, related to an innovative and promising improvement in medical education.
The developed application followed certain characteristics to meet the objectives of proposing an auxiliary tool in the preparation of residents of orthopedics and traumatology for the TEOT exam, based on the principle of retrieval learning. In addition, the method used to create the training application for orthopedic residents was based on some pillars of education by simulation: positive feedback, repetitive practice, difficulty scale, variable clinical catch, individual or group learning and validation of the simulator25. All these pillars were taken into account during the creation of the application assessed in this study. It has been observed that witnessing the development and improvement of their skills throughout the training increases the residents' confidence, as well as receiving constructive, direct and specific feedback26. The developed application allows the user to follow their evolution throughout the test, in addition to receiving the comments related to each question answered by them. The other applications on the market do not have comments related to the questions, do not have the bibliographic reference from which the question was taken, do not allow the comparison of the user's own results and do not allow a comparison of results with the other users.
In the context of the questionnaire applied to the participants, we can infer the high acceptance of the practicality of the application by the sample group, since 96.2% (127) of the users agreed that it is easy to use, whereas 90.7% of the participants affirmed they would like to use the app frequently. The lowest agreement regarding the other evaluations (85.6%, corresponding to 113 users) was related to whether the use of the application was considered as an auxiliary study method for students; this can be justified by the answers of the group of students, since the purpose of the developed application is not to assist, at this moment, the study of the discipline of Orthopedics and Traumatology, but help the residents who will take the Board Exam.
When asked about the usability of the application (part II and III of the questionnaire), the response of the majority of users was positive among the different sample groups.
Regarding the usability, 100% (132) stated that it was a useful technology for the resident’s theoretical learning, in addition to the 93.4% (124) that agreed it is an auxiliary method of learning for general orthopedic physicians. These data indicate the users’ recognition regarding the importance of alternative ways to support the teaching-learning process, as well as references from previous studies, which identified the increase in the number of residents and the limited opportunities to develop surgical and technical skills. Many different solutions have been proposed in education centers around the world, including e-training programs, simulation, and mandatory training to maximize learning opportunities within existing resource constraints27. In this context, the use of simulation applications allows residents to practice skills in a safe and controlled environment, demonstrating their improved confidence and minimizing patient risk.
For comparative purposes, a study related to the development of a teaching and training application in anesthesiology, analyzed 20 participants (medical students and residents) based on the SUS evaluation score of 90.628, which is referred to as a high-value usability index28. Although the SUS® score of our study was slightly lower (84.2), the size of our sample denotes a more significant impact and scientific representativeness, as well as greater variability, being translated by a similar sample reliability between the two studies (0.79). These data were measured taking into account Cronbach's alpha coefficient, which represents the best method to evaluate variation between samples14. These data demonstrate the importance of the SUS® score obtained by the application developed by this study.
Another study assessing a medical education application for residents in the area of surgical simulation was able to conclude that this type of tool is a good complementary alternative for the training of these students’ profile by increasing cognitive performance and thus ensuring greater patient safety29. The Quiz Ortopedia®, albeit through a different proposal, can achieve the same goal by helping residents take the Board Exam after having developed more knowledge of the covered content.
Among the limitations of the study, we can mention the lack of a comparative efficiency test between residents who used the application and the residents who studied using only the traditional methods. Another limitation is that it included residents from only one federation unit, not covering all the other states.
More comparative studies are required, in the near future, to consolidate the efficiency of the Quiz Ortopedia® application as a support teaching and training method aimed at the education of medical residents of orthopedics and traumatology.
CONCLUSION
This study demonstrated the development, use, and evaluation, through a questionnaire validated in the study, of a mobile application created for the iOS® and Android® platforms aimed at the training of residents in orthopedics and traumatology based on the resolution of questions.
The application showed high usability with good sample variability when validated by specialists in Orthopedics and Traumatology, as well as by medical students and orthopedic residents. The developed application was successful in the tests performed and may be an alternative in medical education in the orthopedic area.