texto em 
Inglês (pdf)
Artigo em XML
Tradução automática
Enviar este artigo por email
Permalink
INTRODUCTION
Medical decision-making “at the bedside”, that is, when the patient is present, has been supported by consulting reference material. The advance of information technology (IT), represented in recent years by the widespread use of the internet and mobile devices (smartphones and tablets), makes this process easier, especially considering the exponential growth of medical literature 1 .
This consultation process for decision-making, whether to decide on the best diagnostic method, the best treatment for a certain patient, among other issues, also becomes a form of medical education. This is because, in order to solve problems, the physician or medical student has a stimulus for the study and seeks the best answer to a demand of their practice. Thus, learning becomes more natural, favoring the reflection on the action and the production of new knowledge in a permanent education movement 2 .
Considering the unprecedented phenomenon of knowledge expansion in the medical field, anchored by the advancement of the internet in its dissemination, Evidence-Based Medicine (EBM) emerges as an indispensable tool. EBM allows medical knowledge to be better classified based on the assessment of the quality of recommendations for health practice 3 .
In this context of medical education, there are the Medical Residency Programs (MRPs), a postgraduate modality focused on working in practice scenarios, with moments of theoretical discussion, strongly based on EBM. The Family and Community Medicine (FCM) MRPs stand out for the discussion of EBM tools, taking into account the patient’s perspective in the decision-making process.
Different associations of family and community physicians worldwide have advocated the use of the Strength of Recommendation Taxonomy (SORT) methodology to determine the quality of scientific evidence in the context of EBM, based on the classification of the recommendations 4 . Another complementary method used for evidence classification is the “5S” scale. It defines five different types of sources for searching for medical evidence (studies, syntheses, synopses, summaries, and systems). Summaries and systems are considered to be those of the highest quality, but the use of systems is still limited to some specific scenarios 5 .
Therefore, summaries are classified as the most reliable and most widely available sources of information for health information searches. Summaries are available from different internet-based platforms and accessed from computers and mobile devices. These platforms have become the main sources of information for medical decision-making 6 .
Despite the wide variety of summary platforms available, most of them pose a number of obstacles to the user. These include the cost, as there is usually a subscription fee; the language, since many are available only in the English language and the fact that most of them do not have information compatible with the particular epidemiology of our country. It is of great interest to have a free, evidence-based, fully national medical decision-making tool that can overcome the aforementioned obstacles of the traditional platforms.
Strategies to overcome these obstacles, already accomplished beyond the medical field, are diverse and some of them could be thus applied. Among them is crowdsourcing, which is the construction of a content in a collaborative way by the users 7 . Thus, the user, from a mere consumer of content, becomes the main actor in the process of creation, sharing and regulation of knowledge.
The concept of gamification, which incorporates game elements in non-game scenarios, is another innovative and important strategy to overcome the barriers of traditional summary platforms. Studies have shown that the use of this tool in education can increase user adherence and participation in digital platforms 8 .
Considering that the current summary platforms are available in digital format (computers and other smart devices), ways of assessing how easy it is to use them are also necessary. This is called usability and several validation methodologies and tools are available. Among them is the System Usability Scale (SUS), which is a quantitative tool, measured through a short questionnaire, validated in Portuguese and widely used worldwide, including in healthcare platforms 9 . According to Nielsen, one of the leading researchers in the field, a usability test with 5 users is enough to identify most problems. For tests with distinct groups (i.e., different user profiles using the same software), the tests can be performed with 3 users in each group 10 .
Based on the understanding of the access obstacles in the traditional medical summary platforms, as well as the possibilities of overcoming them, this study proposes the development of a new digital platform for medical decision-making, based on evidence-based medical summaries, gamified, and of which content was created through crowdsourcing, available for access on smartphones, tablets and computers. Also, describe the evaluation of such platform, as well as the recommendations generated in it, in order to reflect on its viability as a consistent tool for medical decision-making.
METHOD
The platform was built from web development tools, using Angular as the framework, which is nothing more than a set of libraries used to facilitate the construction of the application 11 . This choice was made because it is a modern tool, sponsored by a large technology company (Google) and with multiple users worldwide. Moreover, it allows content to be made available on multiple devices, including the web, tablets, and smartphones, among others.
The platform structure consists of a server, responsible for user authentication and access authorization to information. This server communicates with a database, which stores the user registry, the contents created and the evaluation of each content. The server also communicates with Microsoft Azure 12 cloud platform, which is responsible for image storage and application hosting.
In addition to the database, there is an interface responsible for the access of and interaction with information, which is available in the web and as a mobile app version. This interface was developed with web technology (HTML5, CSS and JavaScript).
Figure 1 shows the relationship between users, interface, and database. Therefore, the summary platform consists of a free software, available for access through computers, smartphones and tablets.
The conceptual framework of the platform is based on four axes: collaboration, updating and editing, evaluation and gamification.
Collaboration
It is the main axis of the platform, allowing the user to send the information to be used by others. The user can create three different types of medical content: topics, which are nothing more than evidence-based summaries of diseases, problems, syndromes, laboratory alterations, signs or symptoms; medications, which are summaries of drugs containing their main uses, recommended dosage, pharmacological aspects, adverse effects, and drug interactions, among others; calculators, which allow quick calculations of risk scales, anthropometric measurements, dose calculation, among others.
Updating and editing
Any platform user can, in addition to creating content, update or edit a topic, medication or calculator created by another. Each edition, updating, or modification made is saved in the database and can be retrieved in the interface, preventing the action of vandals that may erase the created content.
Evaluation
As this is a clinical decision-making platform, the quality of the available information is indispensable. There is a mechanism present in each topic and medication, in which the user can rate the content (from 1 to 5). The final score of a topic or medication consists of the average of the evaluations. After each edition or update, the score changes according to the percentage of content change. For example, if a content is modified by 50%, its score is halved. After one edition, all users can reassess the changed summary.
Gamification
Considering the already discussed potential of gamification in the context of digital education, such elements have been incorporated into the platform. The incentive to participate in the platform comes from the distribution of points by the collaboration. For each content created (topic, medication or calculator), the user gets 10 points; for each content edition, the user earns 5 points and for each evaluation, 1 point. The sum of the points earned by the user allows them to advance levels (starting at 1 and ending at level 99). The score required to advance at each level consists of the formula
Guiding axes for platform validation (evaluation part)
After the platform development, the validation process was performed. For that purpose, four axes were taken into consideration: satisfaction survey, usability assessment, study of data generated on the platform and evaluation of the strength of the created recommendations.
This validation process started shortly after the development of a minimally viable version of the platform and involved guest FCM MRPs, with program members (whether coordinators, tutors, preceptors or resident physicians) being the first testing the software and participating in the evaluation steps.
The statistical analyses of the evaluation process were performed using the “R” software 13 .
•Satisfaction survey
The users’ perception regarding the platform was measured from a satisfaction survey, consisting of a semi-structured questionnaire sent to all users registered in the system. This questionnaire served as a guide for measuring overall satisfaction, whether the user recommended the software to someone and considered it a useful tool for clinical decision-making. Additionally, it also had an open field for criticisms and suggestions, used as a guide for improvement.
•Usability evaluation
The evaluation of usability perception by users was based on the random selection of 9 physicians (among preceptors, tutors and residents of the FCM MRPs), which were divided into 3 groups of 3 participants each:
Group 1 – Active users of the platform;
Group 2 – Non-platform users, but who have received specific training for its use;
Group 3 – Non-platform users who did not receive specific training for its use.
Based on the selection of participants, they were asked to perform some predetermined tasks, and the entire progress was recorded using an online tool 14 . The selected participants were in a room, accompanied by a team member, and they were asked to perform the following tasks:
Register on the platform;
Perform the login;
Create a new topic, entitled “Heart Failure” and write “Test 2018” in the content;
Insert the registration of a medication (Dipyrone) and write “Test 2018” in the content;
Create a new calculator (to calculate Body Mass Index - BMI), remembering that BMI = Weight / Height 2 ;
Edit a pre-existing topic (“Syphilis Infection”) and write “Test 2018” anywhere in the text;
Evaluate a pre-existing topic (“Syphilis infection”).
After performing the tasks, the users were asked to answer the SUS questionnaire, in order to measure the system usability. Aiming to evaluate the usability according to the tasks, the number of performed tasks, total performance time and mean and median of SUS score by group were evaluated. Moreover, the SUS means were compared between the groups and age of users in relation to the SUS score.
•Study of platform-generated data
User activity while using the platform was recorded in the software database, longitudinally, over 4 weeks. The analysis considered the following:
Number of registered users;
Number of topics created;
Number of medications created;
Number of calculators created;
Topics with sufficient mean (i.e., with a score greater than or equal to 3 points);
Topics with insufficient mean (i.e., less than 3 points).
These data were analyzed individually and some of them compared, observing the statistical significance of the comparisons. The following were compared: number of registered users in relation to the number of topics created; number of users in relation to the number of medications created; number of users in relation to the number of calculators created; number of users in relation to the number of topics with sufficient mean; number of topics created in relation to the number of topics with sufficient mean.
These indicators aimed to identify mainly whether the increase in the number of users will imply greater participation in the platform and whether this greater participation is reflected in higher quality of the generated content, something essential for its use as a medical education tool.
•Assessment of the Strength of Recommendations
The process of developing a topic or medication in the platform has generated a series of recommendations to be followed by the other users, when they have to make a clinical decision. As a way to facilitate the decision-making process, platform users have been advised to classify each published recommendation using the SORT methodology. Therefore, the validation should ensure that such information is in accordance with the best available evidence.
For that purpose, a random sample of 20% of user-created topics was validated by an external evaluator with expertise in EBM. The recommendations were evaluated based on the SORT methodology, being classified as levels A, B or C, in which A refers to higher quality evidence and C to lower quality. The results of this classification were analyzed individually and also compared with the evaluations provided by the users, aiming to ensure the reliability of the collaborative system as an aid to medical decision-making.
After this testing phase, the platform was finally launched on the web and mobile devices. After the release, both the software and the evaluation tool are being constantly updated and refined.
As this was a research involving human beings, it was submitted to the evaluation of the Research Ethics Committee, and it was approved under CAAE number 00241018.1.0000.5282.
RESULTS AND DISCUSSION
The platform
The developed platform is in its first version. The mobile and web versions have a similar interface, consisting of a side menu with the “Home”, “Topics’, “Medications’ and “Network” buttons, each leading to a different screen.
When accessing the “Topics”, “Medications” and “Calculators” sections, the user has access to a listing of the topics, medications and calculators that have been created, respectively, with their respective authors and also the average rating of each. One can also create new content by clicking the floating button in the bottom right corner. In Figure 2 there is an example screenshot from the “Topics” section.
Figure 3 shows the new topic creation screenshot. Its structure is similar to that of new medication creation, having a text editor available for free content creation.
To create a new calculator, it is necessary to include the variables and also the formula or equation of the calculation to be performed. Figure 4 shows this process for the creation of a BMI calculator.
Finally, accessing a content (topic or medication), one can see its full content. It is also possible to attribute a rating (1 to 5, represented by stars) to it, as shown in Figure 5 .
The “Network” section, in turn, shows the user’s current level, as well as how many points are required to advance to the next level. In the upper right corner, one can also see the user’s level ( Figure 6 ). This comprises the platform’s gamification structure, where each user’s interaction (content creation, editing, and evaluation) is converted into experience points.
The “Network” section also shows the content developed by the user. In the future, it is intended that this section also allows interactions between users, including the possibility of creating groups and discussion forums.
Evaluation results
•Satisfaction survey
The questionnaire was answered by a total of 16 people. Figure 7 shows the participants’ level of schooling. A greater participation of physicians over resident physicians and medical students can be observed, which accounts for more than 50% of the total.
Regarding the participants’ age, the mean was 31 years, with a median of 30.5 (25 - 40) years. The questionnaire, consisting of 4 questions answered on a Likert scale ranging from 0 to 10, has stronger positive answers the closer they are to 10.
As for the results, regarding the users’ satisfaction with the platform, the mean of the answers was 8.9, with a median of 9 (7 - 10). When asked if they would recommend the platform use to someone else, the mean of the answers was 9.6, with a median of 10 (8 - 10). About the usefulness of the platform as a clinical decision-making tool, the mean was 8.1, with a median of 8 (6 - 10). Finally, when asked if they would use the platform again, the mean answer was 9.2, with a median of 10 (7 - 10). The results are shown in Table 1 .
TABLE 1 Results of the User’s satisfaction questionnaire
| Question | Mean | Median |
|---|---|---|
| 1 – Satisfaction with the use of the platform | 8.9 | 9 |
| 2 – Recommends the use of the platform to another person | 9.6 | 10 |
| 3 – Considers it useful as a clinical decision-making tool | 8.1 | 8 |
| 4 – Would use the platform again | 9.2 | 10 |
In the open field intended for criticisms and suggestions, some interesting considerations were observed. Regarding the suggestions, a recurring request was to allow the use of the platform offline, aiming to facilitate consultation in regions without internet connection. A division of content by medical specialty was also suggested, as well as an easier way to generate references for the created content.
Some users praised the gamification system, also suggesting a ranking of users with the highest levels. As criticisms, some users experienced some slowness on the platform, as well as an initial lack of content.
The results indicate a tendency towards good overall satisfaction with the platform use, and users find the platform useful for decision-making, in addition to the interesting possibility of retention due to the good result regarding the repeated use of the platform. Medians with values close to the means also indicate that the results were not influenced by extreme values.
•Usability assessment
Figure 8 and Table 2 show the comparison between the means obtained at the usability test divided by groups. The mean score at the SUS is 68 points, and the result of the 3 groups was classified as A , which is equivalent to an index above the 90 th percentile. This indicates that, on average, the platform has good usability, with no major problems even for users who have never used it. On the other hand, the important difference between the scores of users who have already used the platform, when compared to those who have never used the platform can demonstrate the need for a tutorial or training section, which can be made available within the platform.
TABLE 2 Total score in SUS divided by groups
| Group | Mean | Median | Minimum | Maximum |
|---|---|---|---|---|
| Group 1 | 92.5 | 92.5 | 90 | 95 |
| Group 2 | 87.5 | 87.5 | 85 | 90 |
| Group 3 | 83.3 | 82.5 | 82.5 | 85 |
Legend: SUS – “System usability scale”.
The time to perform tasks divided by groups is shown in Table 3 . A time difference can be observed between the groups, which shows a trend towards the faster accomplishment of tasks among those who have used the platform before or received training. Regarding the fulfillment of tasks, all were performed by all participants of the usability test. Despite the divergence in time, this difference was less than 3 minutes between the extreme groups, not being significant enough to hinder the platform usability. Another relevant issue is that the medians with values very close to the means indicate that extreme values did not significantly influence the mean.
TABLE 3 Total time of task performance by the groups
| Group | Mean | Median | Minimum | Maximum |
|---|---|---|---|---|
| Group 1 | 5min41s | 5min25s | 5min17s | 6min22s |
| Group 2 | 6min40s | 6min37s | 6min12s | 7min08s |
| Group 3 | 8min35s | 8min50s | 7min43s | 9min12s |
The mean age of the users participating in the usability test was 30.4 years, with a median of 31 (25 - 35) years. To compare age and SUS score, Pearson’s coefficient was used, due to the normal distribution of data (verified by the Shapiro-Wilk Test 15 ). Based on the comparison, it was verified that the difference between the SUS score by age is not statistically significant (p> 0.05). This seems to indicate that the platform did not have its usability influenced by the variation in the users’ age range.
•Study of platform-generated data
Table 4 shows the results throughout the four weeks of evaluation focused on the study of platform-generated data. During this period, a total of 19 users registered on the platform and a total of 11 topics, 7 medications, and 1 calculator were created.
TABLE 4 Results of the platform data assessment
| Week 1 | Week 2 | Week 3 | Week 4 | ||
|---|---|---|---|---|---|
| Total users (cumulative) | 7 | 10 | 13 | 19 | |
| Topics | 1 | 2 | 3 | 5 | |
| Medications | 0 | 1 | 2 | 4 | |
| Calculators | 0 | 0 | 0 | 1 | |
| Topics with sufficient mean (cumulative) | 6 | 9 | 13 | 19 | |
| Topics with insufficient mean (cumulative) | 1 | 1 | 0 | 0 | |
The number of users was compared with the number of topics created using Pearson’s coefficient, due to the normal distribution of data (verified by the Shapiro-Wilk test). Based on the use of Pearson’s coefficient, there is a tendency for the number of topics to grow along with the number of users, showing a statistically significant correlation (p <0.05).
Pearson’s coefficient was also used when comparing the number of users and the number of medications created, due to the normal distribution of data (verified by the Shapiro-Wilk Test). In this case there is also a positive trend, indicating a statistically significant correlation (p <0.05).
The Mann-Whitney test was used for the comparison between the number of users and the number of calculators created, due to the non-normal distribution of the data related to the calculators (verified by the Shapiro-Wilk test). A positive correlation was found between these data, pointing to a statistically significant correlation (p <0.05).
Finally, the comparison between users and topics with sufficient mean and between created topics and topics with sufficient mean, Pearson’s coefficient (normal distribution of data verified with the Shapiro-Wilk Test) was also used. As a result, there was a growing trend in both cases, with a statistically significant correlation indication (p <0.05).
These results corroborate the thesis that, with the increase of users, there is a tendency towards the proportional increase in content, as well as an increase in the quality. Thus, expanding the dissemination and encouraging the participation of as many users as possible seems like a feasible strategy for improving the platform.
•Evaluation of the strength of recommendations
Of the 11 topics created, a total of 37 recommendations were generated on the platform. Of these, a sample of 7 recommendations was randomly selected for evaluation by the external validator.
Considering the evaluations performed by the validator, 4 recommendations (57%) were classified as high quality, that is, they obtained recommendation strength A.
Chart 1 shows the results of comparative evaluations between platform users and the validator. Of the total assessed recommendations, 4 of them (57%) obtained the same classification by users and by the validator. One of them was not classified by platform users (14.5%) and 2 of them (28.5%) were classified differently, with one of them receiving a better classification by the experienced validator when compared to that made by the users.
CHART 1 Evaluation of the strength of a sample of generated recommendations
| Recommendation | Topic | Strength of recommendation in the platform | Strength of recommendation by the validator |
|---|---|---|---|
| “Treatment for mild to moderate migraine: Ibuprofen 400 mg” | Migraine in adults | A | A |
| “Pre-adolescents with hypertension must be assessed as soon as possible regarding secondary hypertension.” | Arterial hypertension | B | C |
| “The initial treatment recommended for hypertensive patients with heart failure or previous myocardial infarction includes a beta-blocker and an ACE inhibitor.” | Arterial hypertension | A | A |
| “The combination of increased urinary frequency and dysuria, with no vaginal discharge, is diagnostic for urinary tract infection.” | Urinary tract infection in women | No assessment | C |
| “Initial assessment in suspected heart failure should include clinical history and physical examination, laboratory evaluation, chest radiography, and electrocardiography. The echocardiography can confirm the diagnosis.” | Heart failure | C | C |
| “ACE inhibitors seem to reduce rates of mortality and myocardial infarction and hospitalization in patients with left ventricular dysfunction or symptomatic heart failure.” | Heart failure | A | A |
| “Do not request imaging tests unless red flags are present.” | Back pain | B | A |
The high rate of high-quality recommendations observed in the sample, together with the fact that the users’ evaluations were similar to those made by the validator, point to the hypothesis that the platform may be an important tool for evidence-based medical decision-making. A warning is made to the part of the recommendations that was not classified by the users, requiring a longitudinal observation of the use of the platform and increased participation to verify whether this sample will be representative of the whole.
FINAL CONSIDERATIONS
The platform development allowed establishing an important interprofessional relationship between the IT and healthcare areas. It allowed the understanding, research and development of technological tools to aid medical decision-making. Possibilities have also been observed that point to the proposed crowdsourcing platform as a viable way to consolidate a medical summary platform in Brazil. The perspective of overcoming the barriers of traditional platforms by gathering features not usually found in other systems (such as collaboration, gamification, evaluation, and reputation) makes this proposal a groundbreaking one.
The evaluations performed showed that the increase in the number of users is capable of converting into a greater content production and consequent increase of its quality. Moreover, usability tests have shown encouraging results, even for users who had never used the platform before. Comparing the level of evidence attributed by users to that of an expert, the results were similar, pointing to the reliability of the collaborative platform as a medical decision-making tool. Regarding the gamification, previous studies on the topic and the platform users’ perception indicate that this is a good strategy, not only to add new users, but also to increase their retention as active members.
Additionally, the work allowed a reflection on the educational processes. By contributing to content creation and consulting information for problem-solving practice, physicians and medical students are expected to experience a differentiated model of medical education. Based on this model, learning from real-life problems and the construction of collaborative knowledge are integrated.
As future works, in addition to the inclusion of improvements obtained from the evaluations, new assessments focusing on user satisfaction and, mainly, on the quality of evidence of the generated recommendations may be carried out. Finally, it will be possible to count on the wide adhesion of users, which will define the reach and success of the platform and the strengthening of evidence-based decision-making in the medical area.
