texto em 
Inglês (pdf)
Artigo em XML
Tradução automática
Enviar este artigo por email
Permalink
INTRODUCTION
The purpose of the teaching methods is to provide an environment that facilitates student learning. In this context, motivation is directly related to the efforts made by a certain person to acquire knowledge or develop a certain skill. Highly motivated students actively engage in studies without worrying about external rewards. On the other hand, when poorly motivated, they may have the skills necessary for learning and not achieve good proficiency1. Thus, instruments based on solid theories, which assess motivation, after exposure to different teaching strategies and materials, can contribute to the analysis and decision about their effectiveness.
Several theories have been conceived to explain the factors and phenomena that interfere with the motivation for learning, such as intrinsic and extrinsic motivation theory, self-determination, social cognitive theory, the expectancy-value theory, as well as the ARCS (Attention, Relevance, Confidence and Satisfaction) model, which supports the construct of the Instructional Materials Motivation Survey (IMMS) instrument, analyzed in the present study2),(3.
The ARCS model of motivational design was proposed by Keller3, based on cognitivist perspectives. The author’s objective was to find more effective ways to understand the main influences of motivation on learning and create a systematized method to identify and solve the associated problems, building a bridge between the study of motivation and instructional practice, aiming to increase student motivation. Its origin is based on the expectancy-value theory which, in turn, establishes that people become motivated to engage in an activity, if it is associated with the satisfaction of their personal needs (values) and a positive expectancy of success in its performance4)-(6.
In Keller’s3 original motivational design model, these two dimensions (Value and Expectancy)5),(6 were expanded to four, as follows:
Value was subdivided into two dimensions: Interest and Relevance;
The Interest dimension was more precisely related to factors associated with attention in the educational environment.
Relevance was linked to activities directed by objectives or goals.
The Expectancy dimension was maintained, being related to the student’s belief that they would be successful in the task; and the Result dimension was created, which represented the value of reinforcement provided by the instruction, as explained by operant conditioning.
Based on this initial construct (Interest, Relevance, Expectancy, Result), qualitative experiments were conducted to validate the model and then it was modified to the current ARCS (Attention, Relevance, Confidence, Expectation) model. The dimensions of the ARCS model are defined as follows by the author3),(4: Attention presents characteristics that stimulate and sustain interest and curiosity; Relevance is related to the students’ personal needs and goals, generating a positive learning attitude; Confidence is related to the students’ beliefs and feelings, so that they can be in control of their learning result and will succeed; Satisfaction is generated by the process and/or result of the learning experience and may be related to the intrinsic (sense of competence) or extrinsic (academic degrees, certificates, material reward) factors and the desire to continue learning.
Based on the ARCS theoretical construct, the Instructional Materials Motivation Survey (IMMS) instrument was developed to assess the motivation of students exposed to educational materials or strategies4. As a situational instrument, the IMMS does not measure levels of motivation in a generalized way, such as, for instance, motivation regarding school learning. The purpose of this instrument is to measure how much students are motivated for a certain instructional activity, such as video classes, or for a specific strategy such as, for instance, remote learning. Due to the importance of being well interpreted by the respondent, it can be used in populations of students as early as those attending high school14.
According to Keller4, his conception was based on the construction of a pool of items based on ARCS motivational concepts, which were reviewed by 10 undergraduate students well acquainted with motivational literature. Based on the students’ answers, the items were revised and retested or excluded, thus originating the set of variables that currently constitute the IMMS in its four dimensions: Attention, Relevance, Confidence and Satisfaction. Its reliability, measured through internal consistency using Cronbach’s alpha coefficient, was 0.96 for the full scale (Attention = 0.89; Relevance 0.81; Confidence 0.90; Satisfaction 0.92), according to Keller4.
Since its conception, the IMMS has been widely used, in several countries, in different teaching-learning contexts, especially in those using technological means in the pedagogical platform, such as web-based courses, e-learning and gamification7-10. Due to the robustness of the ARCS construct, the wide employability of the IMMS to assess student motivation and the need for an appropriate instrument to perform this measurement in Brazil, Cardoso-Júnior et al.11 carried out the translation and cross-cultural adaptation of the IMMS into Brazilian Portuguese.
However, despite being widely used since its creation to measure student motivation, with the aim of improving teaching strategies, the studies that evaluated the psychometric validity of the IMMS showed heterogeneous results12)-(14. These studies resulted in the reduction of the number of items and the modification of their distribution into dimensions, emphasizing the instrument’s situational characteristic, as demonstrated by Keller15, and emphasizing the need to modify the instrument to accommodate the context of its application. In other words, it is suggested that the instrument’s situation is dynamic and varies according to factors related to its implementation, ranging from the students’ characteristics to those of the involved instructional programs.
In this sense, Huang et al.12 evaluated the validity of the IMMS using first-year engineering students studying a computer-based tutorial called MATLAB. The result of the validation was the maintenance of 20 items, distributed into four dimensions, differently from the original instrument. In turn, Loorbach et al.13 conducted a study in a self-directed learning environment. They concluded that, in this context, the instrument could be reduced to 12 items that adequately fit the four original dimensions of the IMMS, thus proposing an instrument called the Reduced Instructional Materials Motivation Survey (RIMMS). Recently, Hauze et al.14 studied the IMMS validity in a simulated training scenario applied to nursing students. In this study, 19 items were maintained, distributed into the four dimensions of the original IMMS. In conclusion, they emphasized that the validity of its use in other populations and to evaluate other types of instructional methods cannot be predicted by the study.
Based on the analysis of these studies, a lack of homogeneity is observed in relation to the results of the evidence of the IMMS construct validity, when applied in different contexts, related to the students’ characteristics, the evaluated instructional material, the teaching environment and the sociocultural context. Therefore, considering the heterogeneity of the results of the described studies, their situational characteristic and the importance of investigating their applicability to measure medical students’ motivation in the context of remote teaching of basic sciences, this study aimed to analyze the evidence of the IMMS construct validity, previously translated and adapted into Brazilian Portuguese, applied to students attending the first two years of the undergraduate medical course at UNIFENAS (Campus Belo Horizonte), to assess their motivation in relation to asynchronous video classes of human anatomy, attended during the social isolation period due to the COVID-19 pandemic.
The increasing use of this instrument in technological teaching environments, as observed in the literature, emphasizes the importance of having a version with valid parameters known in Brazil, since the use of new technologies in the field of education is a growing trend, which needs evaluation of its motivational properties7),(9. In addition to its applicability in this context, the IMMS also shows as a particular feature, in relation to other instruments that assess motivation for learning, the fact that it is based on the ARCS model of motivational design, whose concepts and strategies can be applied, based on the diagnosis made through the observation of the results presented by the instrument, for correction and improvement of the instructional activity under investigation4.
METHODS
Study design
This was a cross-sectional study that evaluated the validity evidence of the Instructional Materials Motivation Survey (IMMS) questionnaire, using a version previously submitted to translation and cross-cultural adaptation11.
Sample characteristics
The target population consisted of students of any gender or age, attending the first two years of the undergraduate medical course at Universidade José do Rosário Vellano - UNIFENAS, Belo Horizonte campus. The study included, in a convenience sampling, all students who were regularly enrolled in the first semester of 2020, in the Human Anatomy Laboratory Practices (LP) strategy, who wished to participate and did not meet the exclusion criteria. The exclusion criteria were: those who declared they wished to leave the study, lack of adherence to the protocol, prior knowledge of the IMMS, allocation error in relation to the research criteria, and foreign nationality. After applying the inclusion and exclusion criteria, the students who were interested in the study signed the Free and Informed Consent Form, being allocated to the study. This research was approved by the Research Ethics Committee - CEP/UNIFENAS as stated in the Certificate of Presentation for Ethical Approval (CAAE), under number 31795820.4.0000.5143, Opinion 4.074,707.
Asynchronous human anatomy video lessons
The undergraduate medical course at UNIFENAS-BH, founded in 2003, uses the Problem-Based Learning methodology. Human anatomy is studied throughout four semesters, in the first two years. For the teaching of anatomy, synthetic models, a digital anatomical table, imaging exams, videos of surgical procedures and prosection are used. The curriculum consists of thematic blocks that carry out the horizontal and vertical knowledge integration. The course lasts 12 semesters divided into pre-clinical, clinical and internship phases.
During the social isolation period, which occurred as of mid-March 2020, it was decided to use asynchronous video classes to maintain the Human Anatomy educational strategy. The five tutors of this teaching strategy recorded the classes using the Open Broadcaster Software (OBS)® program, version 25.0.4, and uploaded them to the Google Drive® Institutional application, with the weekly access link available in the virtual learning environment (Moodle®), according to the schedule of the thematic blocks. The mean duration of the video lessons comprised 30 minutes. After being included into the Institutional Google Drive®, the video classes were available to students for consultation, on demand, throughout the entire ongoing thematic block.
Application of the Instructional Materials Motivation Survey (IMMS) questionnaire
The original IMMS is derived from the ARCS construct, having been developed with the intention of being applied after students have had an educational experience. The questionnaire consists of 3 sentences related to the instructions for completion and 36 items belonging to the four dimensions of motivation: Attention (12 items); Relevance (9 items); Confidence (9 items) and Satisfaction (6 items). Its purpose is to measure the respondent’s motivation, using the Likert scale, with the following scores: (1) totally disagree; (2) partially disagree; (3) neither disagree nor agree; (4) partially agree; (5) totally agree. The total score ranges from 36 to 180 points, and each dimension or the total score can be evaluated, thus generating the motivation score. The minimum, maximum and mean values for each dimension vary, as the number of items is different and they are independent. Some items, mentioned as inverse ones, should be scored as follows: 5=1; 4=2; 3=3; 2=4 and 1=5. Higher scores in the total score indicate greater motivation15.
The IMMS used in this research was the version that was previously translated and adapted into Brazilian Portuguese by Cardoso-Júnior et al.11, which maintained all dimensions and items of the original questionnaire. The self-administered instrument was created using the Google Forms® application, preceded by the informed consent and the sociodemographic questionnaire. The access link was posted on the Moodle-UNIFENAS platform. After reading the consent form and accepting to participate in the study, the student was directed to the questionnaire.
As part of the information about the filling out of the questionnaire, it was clear that the instrument was intended to “... assess your motivation in relation to the Human Anatomy video classes, studied in Laboratory Practices (LP) ...” For that purpose, words related to the instructional material were adapted to this context of application in the items’ sentences, according to the IMMS application instructions, without changing their meaning15.
Students who, after reading the consent form, checked the option “I don’t agree” were directed to a message thanking them for their attention, not having access to the questionnaires. The mean time to complete the forms was 15 minutes.
Principal component analysis
Aiming to verify the evidence of the studied construct validity, a statistical analysis was performed using the principal component analysis (PCA) extraction method and Varimax rotation. The use of the rotation aims to facilitate the interpretation of the dimensions, that is, the saturation of items in the components, finding a solution in which each variable shows high saturation in a few components16),(17. Items with an inverse direction had their scores inverted.
For PCA, the sample size should be ≥100 observations, preferably reaching five or more observations per instrument item, to avoid extracting sample-specific components, which are not very generalizable17. In this study, the sample size (n=211) was adequate, comprising six subjects per IMMS item.
To assess the convenience of the PCA model, the Bartlett test of sphericity and the Kaiser-Meyer-Olkin (KMO) sample adequacy measure were used. Values > 0.5 in the KMO test and p<0.05 in the sphericity test were defined as parameters for the PCA to be considered adequate for the set of studied variables.
The establishment of the number of retained dimensions was performed through determination techniques based on eigenvalues. Eigenvalues > 1 were considered as a parameter (Kaiser-Guttman criterion). The loads (item saturation) were considered significant when they were greater than 0.40 (parameter considered adequate for a sample of 211 subjects at a significance level of 0.05 and 80% power)17. Measures related to the method without rotation were used, followed by the VARIMAX rotation, as performed in previous studies, which allows for a better comparison of results12),(14.
Measures of commonality (MOC) and measures of sample adequacy were also performed for each of the analyzed items, considering adequate values for commonality and for sample adequacy (Measures of Sampling Adequacy - MSA) > 0.50. Aiming to assess the interdependence of the latent dimensions identified by the PCA, Pearson’s Correlation analysis was performed.
Additionally, the differences between the observed correlations (correlation matrix with all original items) and the reproduced correlations (correlations estimated by the construct matrix) were assessed, where such differences, called residuals, indicate a good adjustment of the generated model, when they show very low values.
The parameters adopted in this study are based on publications by Landis and Koch16 and Hair Jr et al.17. The statistical software SPSS (IBM), version 17.0, was used to perform the analyses.
RESULTS
A total of 211 students aged 17 to 46 years, with a mean age of 21.4 years, participated in this study. Of these, 33% were aged between 17 and 19 years, 44.5% between 20 and 22 years, 13% between 23 and 25 years, 6% between 26 and 30 years old and 3.5% aged at least 31 years. There was a predominance of women (67.8%). In relation to the course period, 46.4% of the students were attending the first period, 35.1% the third period and 18.5% the fourth period. The students attending the second-period were excluded from the study because, unlike the others, they participated in synchronous online classes. Only 5.7% of students had another university degree.
Analysis of the IMMS Construct Validity Evidence
The initial PCA was performed with all 36 items that constitute the original IMMS questionnaire, translated and adapted into Brazilian Portuguese11. Table 1 shows the evaluation of the students’ behavior towards each of the IMMS items.
Table 1 Assessment of students’ behavior towards each of the IMMS questions.
| Question | Answers | Total | Mean | SD | ||||
|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||||
| 1. When I watched these video lessons for the first time, I had the impression that it would be easy for me. | 11.4 | 19.9 | 25.6 | 29.4 | 13.7 | 100.0 | 3.1 | 1.2 |
| 2. There was something interesting at the beginning of these activities that caught my attention | 9.5 | 14.7 | 28.0 | 32.2 | 15.6 | 100.0 | 3.3 | 1.2 |
| 3. These video lessons were harder to understand than I would have liked. | 20.4 | 28.4 | 14.2 | 26.6 | 10.4 | 100.0 | 2.8 | 1.3 |
| 4. After listening to the initial information, I felt confident that I knew what was expected to be learned from this activity. | 12.8 | 19.4 | 25.6 | 27.5 | 14.7 | 100.0 | 3.1 | 1.2 |
| 5. Completing the exercises in this activity gave me a satisfying sense of accomplishment. | 10.4 | 15.2 | 18.5 | 34.1 | 21.8 | 100.0 | 3.4 | 1.3 |
| 6. It is clear to me how the content of this material is related to knowledge I already have. | 7.1 | 15.2 | 32.7 | 29.8 | 15.2 | 100.0 | 3.3 | 1.1 |
| 7. Many slides had so much information that it was difficult to choose and remember the important points. | 27.0 | 25.1 | 18.0 | 18.5 | 11.4 | 100.0 | 2.6 | 1.4 |
| 8. These video lessons are interesting. | 10.0 | 11.4 | 18.5 | 32.2 | 27.9 | 100.0 | 3.6 | 1.3 |
| 9. There were stories, pictures or examples that showed me how these video lessons could be important to some people. | 5.2 | 8.5 | 22.3 | 33.2 | 30.8 | 100.0 | 3.8 | 1.1 |
| 10. Successfully completing this activity was important to me. | 4.3 | 4.3 | 21.8 | 27.0 | 42.6 | 100.0 | 4.0 | 1.1 |
| 11. Video quality helped to keep my attention. | 7.1 | 13.3 | 16.1 | 31.7 | 31.8 | 100.0 | 3.7 | 1.2 |
| 12. These video lessons are so abstract that it was hard to keep my attention on them. | 25.1 | 28.4 | 15.7 | 18.5 | 12.3 | 100.0 | 2.6 | 1.4 |
| 13. As I worked on this activity, I was confident that I could learn the content. | 8.0 | 16.6 | 26.1 | 32.2 | 17.1 | 100.0 | 3.3 | 1.2 |
| 14. I enjoyed these video lessons so much that I would like to know more about this subject. | 15.2 | 22.7 | 27.5 | 25.1 | 9.5 | 100.0 | 2.9 | 1.2 |
| 15. The slides in this material look uninteresting. | 30.8 | 31.8 | 16.1 | 12.8 | 8.5 | 100.0 | 2.4 | 1.3 |
| 16. The content of these video lessons is relevant to my interests. | 1.4 | 3.3 | 10.0 | 27.5 | 57.8 | 100.0 | 4.4 | 0.9 |
| 17. The way the information is organized on the slides helped keep my attention. | 8.5 | 12.8 | 19.9 | 34.1 | 24.7 | 100.0 | 3.5 | 1.2 |
| 18. There are explanations or examples of how people use the knowledge of these video lessons | 6.2 | 13.7 | 30.8 | 21.8 | 27.5 | 100.0 | 3.5 | 1.2 |
| 19. The exercises in these video lessons were very difficult. | 18.5 | 33.6 | 38.9 | 7.6 | 1.4 | 100.0 | 2.4 | 0.9 |
| 20. These video lessons have elements that stimulated my curiosity. | 8.0 | 17.5 | 20.4 | 37.0 | 17.1 | 100.0 | 3.4 | 1.2 |
| 21. I really enjoyed studying this activity. | 11.4 | 18.9 | 25.6 | 28.9 | 15.2 | 100.0 | 3.2 | 1.2 |
| 22. The amount of repetition in these video lessons made me eventually bored. | 27.0 | 20.4 | 23.7 | 19.4 | 9.5 | 100.0 | 2.6 | 1.3 |
| 23. The content and style of the slides in this material give the impression that it is worth getting to know them. | 6.6 | 15.2 | 20.4 | 37.4 | 20.4 | 100.0 | 3.5 | 1.2 |
| 24. I learned some things that were surprising or unexpected | 6.2 | 11.8 | 17.1 | 38.4 | 26.5 | 100.0 | 3.7 | 1.2 |
| 25. After working on this activity for some time, I was confident that I would be able to pass a test on it. | 13.8 | 18.5 | 23.2 | 28.4 | 16.1 | 100.0 | 3.1 | 1.3 |
| 26. These video lessons were not relevant to my needs because I already knew most of them. | 63.0 | 29.4 | 4.7 | 1.9 | 1.0 | 100.0 | 1.5 | 0.8 |
| 27. The way feedback was given after the exercises or other comments on the activity helped me feel rewarded for my effort. | 19.9 | 18.5 | 30.8 | 19.9 | 10.9 | 100.0 | 2.8 | 1.3 |
| 28. The variety of reading excerpts, exercises, illustrations, etc., helped keep my attention on the activity. | 10.5 | 16.1 | 21.3 | 32.2 | 19.9 | 100.0 | 3.4 | 1.3 |
| 29. The writing style is boring. | 27.9 | 24.2 | 23.2 | 17.1 | 7.6 | 100.0 | 2.5 | 1.3 |
| 30. I was able to correlate the content of these video lessons to things I have seen, done or thought about in my own life. | 8.0 | 16.1 | 27.5 | 31.8 | 16.6 | 100.0 | 3.3 | 1.2 |
| 31. There are so many words on each slide that it is annoying. | 47.4 | 28.0 | 13.7 | 8.5 | 2.4 | 100.0 | 1.9 | 1.1 |
| 32. It was good to successfully complete this activity. | 4.2 | 3.8 | 22.3 | 30.8 | 38.9 | 100.0 | 4.0 | 1.1 |
| 33. The content of these video lessons will be useful to me. | 0.5 | 3.3 | 7.6 | 30.3 | 58.3 | 100.0 | 4.4 | 0.8 |
| 34. I really did not understand much of the material in these video lessons. | 23.2 | 29.4 | 16.6 | 20.8 | 10.0 | 100.0 | 2.6 | 1.3 |
| 35. Good content organization helped me make sure I learned this material. | 5.7 | 16.1 | 29.4 | 35.1 | 13.7 | 100.0 | 3.4 | 1.1 |
| 36. It was a pleasure working on such a well-planned activity. | 12.8 | 16.6 | 25.6 | 24.2 | 20.8 | 100.0 | 3.2 | 1.3 |
SD: standard deviation.
Note: The wording of the items underwent minor changes to adapt to the instructional activity evaluated (video lessons), without changing its meaning.
The initial model showed the convenience of applying this statistical technique in terms of its adequacy (Kaiser-Meyer-Olkin ENT#091;KMOENT#093; sample adequacy measure) =0.943; Bartlett’s sphericity test p < 0.0001). Table 2 shows the result of the number of factors extracted by the initial PCA.
Table 2 Result of the number of factors extracted based on the initial Principal Component Analysis.
| FACTORS | Sum of squares of factorial loads WITHOUT Rotation | Sum of squares of factorial loads WITH Rotation | ||||
|---|---|---|---|---|---|---|
| Eigenvalue | % of variance | % accumulated | Eigenvalue | % of variance | % accumulated | |
| A | 14,500 | 40,279 | 40,279 | 5,767 | 16,021 | 16,021 |
| B | 1,852 | 5,145 | 45,424 | 5,379 | 14,942 | 30,963 |
| C | 1,698 | 4,717 | 50,141 | 2,954 | 8,206 | 39,169 |
| D | 1,325 | 3,681 | 53,822 | 2,700 | 7,501 | 46,669 |
| E | 1,194 | 3,317 | 57,139 | 2,626 | 7,295 | 53,965 |
| F | 1,081 | 3,003 | 60,142 | 1,943 | 5,398 | 59,363 |
| G | 1,015 | 2,818 | 62,960 | 1,295 | 3,598 | 62,960 |
| Total | 22,665 | 62,961 | -------- | 22,664 | 62,961 | --------- |
Database: 211 students. Rotation Method: Varimax.
However, during the PCA rounds, 11 items were removed from the original instrument for different reasons:
Items 6, 16, 18, 30 and 33 for presenting a low proportion of explained variance (commonality < 0.50) in the initial or subsequent PCA;
Items 10 and 32 due to similar loads shown in various components (dimensions);
Item 19 because it was presented as a component with a single item, that is, it did not show significant correlations with the other variables.
Items 1, 2 and 26 for showing low reliability of internal consistency in the dimensions to which they were allocated.
Table 3 shows the items taken from the original IMMS and the causes of exclusion.
Table 3 Items extracted from the original IMMS and causes of exclusion.
| IMMS Dimension | Excluded Item | Cause of exclusion |
|---|---|---|
| Attention | 02 - There was something interesting at the beginning of these activities that got my attention. | IC |
| Relevance | 06 - It is clear to me how the content of this material is related to things I already know. | MOC < 0.5 |
| 10 - Successfully completing this lesson was important to me. | SL | |
| 16 - The content of these video lessons is relevant to my interests. | MOC < 0.5 | |
| 18 - There are explanations or examples of how people use the knowledge of these video lessons. | MOC < 0.5 | |
| 26 - These video lessons were not relevant to my needs because I already knew most of them. | IC | |
| 30 - I was able to correlate the content of these video lessons with things I have seen, done or thought about my own life. | MOC < 0.5 | |
| 33 - The content of these video lessons will be useful to me. | MOC < 0.5 | |
| Confidence | 01 - When I first watched these video lessons, I had the impression that it would be easy for me. | IC |
| 19 - The exercises in these video lesson were too difficult. | SIC | |
| Satisfaction | 32 - It felt good to successfully complete this lesson. | SL |
IMMS: Instructional Materials Motivation Survey translated and cross-culturally adapted to Brazilian Portuguese; MOC: Measure of Commonality; SL: similar loads in several components; SIC: Single-item component; IC: low internal consistency in the component.
Consequently, the final PCA model was performed after the removal of these 11 items, which were considered inadequate. In this model, the sample adequacy measure (KMO=0.947) shows convenience for the use of the PCA. Furthermore, the Bartlett test shows that there is a set of significant correlations (p < 0.0001) between the 25 remaining items of the questionnaire. Thus, based on these assumptions, the statistical technique of PCA can be applied. Therefore, based on the Kaiser-Guttman criterion (eigenvalues > 1), 4 components were extracted, as shown in the scree plot illustrated in Figure 1 and in the results depicted in Table 4.
Table 4 Results of the final model of the IMMS-BRV Principal Component Analysis.
| Items*/dimensions | Loads (saturation of items in the dimensions) | MOC | MSA | Mean | |||
|---|---|---|---|---|---|---|---|
| A | B | C | D | ||||
| 3 | 0.753 | 0.678 | 0.894 | 2.8 | |||
| 4 | 0.661 | 0.608 | 0.959 | 3.1 | |||
| 5 | 0.529 | 0.503 | 0.954 | 3.4 | |||
| 7 | 0.717 | 0.645 | 0.871 | 2.6 | |||
| 8 | 0.779 | 0.728 | 0.960 | 3.6 | |||
| 9 | 0.589 | 0.503 | 0.958 | 3.8 | |||
| 11 | 0.663 | 0.625 | 0.939 | 3.7 | |||
| 12 | 0.569 | 0.621 | 0.939 | 2.6 | |||
| 13 | 0.734 | 0.691 | 0.958 | 3.3 | |||
| 14 | 0.625 | 0.614 | 0.955 | 2.9 | |||
| 15 | 0.599 | 0.581 | 0.948 | 2.4 | |||
| 17 | 0.551 | 0.621 | 0.963 | 3.5 | |||
| 20 | 0.586 | 0.704 | 0.958 | 3.4 | |||
| 21 | 0.555 | 0.686 | 0.961 | 3.2 | |||
| 22 | 0.631 | 0.560 | 0.939 | 2.6 | |||
| 23 | 0.671 | 0.630 | 0.973 | 3.5 | |||
| 24 | 0.627 | 0.579 | 0.940 | 3.7 | |||
| 25 | 0.790 | 0.700 | 0.930 | 3.1 | |||
| 27 | 0.556 | 0.510 | 0.965 | 2.8 | |||
| 28 | 0.709 | 0.625 | 0.945 | 3.4 | |||
| 29 | 0.693 | 0.626 | 0.918 | 2.5 | |||
| 31 | 0.776 | 0.661 | 0.910 | 1.9 | |||
| 34 | 0.595 | 0.678 | 0.946 | 2.6 | |||
| 35 | 0.533 | 0.642 | 0.938 | 3.4 | |||
| 36 | 0.622 | 0.764 | 0.950 | 3.2 | |||
| Cronbach’s Alpha | 0.93 | 0.87 | 0.76 | 0.78 | global MSA 0.947 | ||
| % of explained variance | 24.299 | 17.416 | 12.149 | 9.256 | |||
| Eigenvalue | 6.075 | 4.354 | 3.037 | 2.314 | |||
| Total explained variance | 63.12% | ||||||
* Items numbered as in the IMMS - version translated into Brazilian Portuguese (Cardoso-Júnior et al.11).
MOC = measure of commonality; MSA = Measure of Sampling Adequacy.
NOTE: The fit of the model was verified by examining the differences between the observed correlations and the reproduced correlations obtained from the reproduced correlation matrix. It is noteworthy that there were few cases (< 10%) with residuals considered “large”; therefore, the obtained model was considered to be valid.
The results of the final Principal Component Analysis model generated the modified alternative version of the IMMS (IMMS-BRV). Thus, the loads (item saturation in the dimensions), commonalities, measures of sample adequacy (MSA), explained variances, eigenvalues and Cronbach’s alpha coefficients of the IMMS-BRV are summarized in Table 4.
The composition of the 4 latent dimensions generated by PCA with Varimax rotation is shown in Table 4. The coefficients (saturation of loads in the dimensions) represent the levels of correlations between the dimensions with each of the variables (items) that constitute them. Therefore, higher absolute values indicate that the dimensions and their respective variables are closely related. Moreover, differences between the observed correlations (correlation matrix with all original items) and reproduced correlations (correlations estimated by the dimension matrix) were examined. These differences, called residuals, showed very low values, confirming a good fit of the generated model.
Therefore, based on these results, we will work with 4 dimensions that encompass blocks of items capable of explaining dimensions of motivation in the face of asynchronous video lessons on human anatomy, according to the medical students’ opinion, instead of analyzing 25 items individually. The total score generated by the sum of the values assigned to the 25 items represents the overall motivation score.
In this study, significant correlations (p < 0.05) were observed between all dimensions generated by PCA. However, the correlation observed between the Interest and Confidence dimensions showed a value > 0.80, showing that they are intrinsically linked, and may be measuring similar concepts of motivation. Table 5 shows the correlation analysis between the four components generated by the Principal Component Analysis.
Table 5 Analysis of correlation between the four components generated by the Principal Component Analysis.
| Dimension/Pearson | Dimension | ||||
|---|---|---|---|---|---|
| A | B | C | D | ||
| A | r | 1.00 | 0.81 | 0.55 | 0.65 |
| p | < 0.001 | < 0.001 | < 0.001 | < 0.001 | |
| B | r | --------- | 1.00 | 0.50 | 0.64 |
| p | < 0.001 | < 0.001 | < 0.001 | ||
| C | r | --------- | --------- | 1.00 | 0.50 |
| p | < 0.001 | < 0.001 | |||
| D | r | --------- | --------- | --------- | 1.00 |
| p | < 0.001 | ||||
Database: 211 students.
r: Pearson’s correlation coefficient; p: probability of significance of the correlation analysis.
IMMS-BRV reliability assessment
Table 6 shows the reliability measures of the IMMS-BRV internal consistency.
Table 6 IMMS-BRV Internal Consistency Reliability Measures.
| Formed dimensions / Items | Cronbach’s alpha if item were removed |
|---|---|
| Interest (Cronbach’s alpha = 0.93) | |
| 8 | 0.92 |
| 9 | 0.93 |
| 11 | 0.93 |
| 14 | 0.92 |
| 15 | 0.93 |
| 17 | 0.92 |
| 20 | 0.92 |
| 23 | 0.92 |
| 24 | 0.93 |
| 27 | 0.93 |
| 28 | 0.92 |
| 36 | 0.92 |
| Confidence (Cronbach’s Alpha = 0.87) | |
| 4 | 0.86 |
| 5 | 0.86 |
| 13 | 0.84 |
| 21 | 0.85 |
| 25 | 0.85 |
| 35 | 0.85 |
| Attention (Cronbach’s Alpha = 0.76) | |
| 7 | 0.75 |
| 22 | 0.71 |
| 29 | 0.67 |
| 31 | 0.68 |
| Expectancy (Cronbach’s Alpha = 0.78) | |
| 3 | 0.71 |
| 12 | 0.74 |
| 34 | 0.66 |
Database: 211 students.
Cronbach’s alpha coefficient was used to assess the reliability of the internal consistency of the items that constitute each of the 4 dimensions generated by the PCA. As observed, all of them showed Cronbach’s Alpha measures > 0.75, indicating good internal consistency of the IMMS-BRV dimensions. Moreover, item removal does not increase Cronbach’s alpha coefficient.
Modified alternative Brazilian version of the IMMS (IMMS-BRV)
When comparing the original version of the IMMS and the results of the PCA performed in this study (IMMS-BRV), it is observed that 4 dimensions were maintained and 11 items were removed. The 25 items that were maintained were redistributed into 4 dimensions, in a different combination than the original instrument. Thus, the names of the dimensions were revised aiming to better explain the items allocated in the same dimension. For this purpose, in the content analysis, the semantics of the items combined with the theoretical reference of the ARCS construct and the expectancy-value theory were observed3),(5.
Thus, once the PCA process was completed, the final structure of the IMMS-BRV was defined, which now has 25 items, without changing the Likert scale originally used in the IMMS. The items were allocated, according to the PCA, into 4 dimensions (A, B, C and D), named as follows:
Interest (explained variance = 24.3%; Cronbach’s alpha = 0.93): the set of meanings of the assertions allocated in this dimension involve items from the following dimensions of the original IMMS: attention (58.3%); satisfaction (25.0%) and relevance (16.7%). The semantic analysis of the meanings of these items allows their aggregation under the Interest dimension, since all of them are clearly related to the respondents’ interest in the evaluated instructional material. The substitution of item nouns by the noun interest itself shows the semantic adequacy of the name given to this dimension. According to the Oxford dictionary Languages18, from Oxford University Press, interest is: “what is important, useful or advantageous, morally, socially or materially.” Also: “state of mind that one has towards what is considered worthy of attention.” In turn, in the same dictionary, relevance is defined as “that which stands out in a comparative or value scale; importance, significance.” Finally, satisfaction is: “...pleasure arising from the realization of what is expected, what is desired.” In other words: ... what one is interested in.
Confidence (explained variance = 17.4%; Cronbach’s alpha = 0.87): in the Confidence dimension, 4 (66.6%) items from the same dimension of the original IMMS were agglutinated. The other 2 items (33.3%) come from the original Satisfaction dimension. These two items are intrinsically related to the satisfaction generated by the confidence in learning, by the expectancy of success. In the very definition of the theoretical frameworks of the ARCS construct, from which the IMMS derives, the satisfaction dimension is seen as the result of the interaction between the three others (Attention, Relevance and Confidence). Thus, in the PCA of this study, the items related to satisfaction were agglutinated in different dimensions according to their semantic affinity with each set of items. Therefore, due to the predominance of items related to the confidence dimension of the original IMMS and the intricate confidence/satisfaction correlation, the designation of this dimension as Confidence in the IMMS-BRV was maintained.
Attention (explained variance = 12.1%; Cronbach’s alpha = 0.76): in this dimension, 3 items (75%) are derived from the original IMMS Attention dimension. Only 1(25%) item derives from the original Confidence dimension. Although originally allocated to the Confidence dimension, its semantic analysis demonstrates it is a statement whose interpretation is precisely in line with the Attention dimension. Therefore, the denomination was maintained.
Expectancy (explained variance = 9.3%; Cronbach’s alpha = 0.78): the 3 items distributed in this dimension derive from the Confidence (66.7%) and Attention (33.3%) dimensions. The semantic analysis of the three items shows that they are intrinsically related to the expectancy (expectation) of the occurrence or not of the understanding, therefore, of meaningful learning. Thus, the set of its meaning allows the aggregation of the term Expectancy to this dimension.
It is opportune to emphasize that the IMMS-BRV, as well as the IMMS, can be adapted to suit specific situations. In other words, the “standard” wording of the items contains more generic phrases such as “this course”, “this class” or “many pages”. They must be changed to suit the specific situation being evaluated, such as “this lecture”, “this video lesson” or “many slides”4),(11.
The modified alternative Brazilian version of the Instructional Materials Motivation Survey (IMMM-BRV) is shown in Chart 1. Although the version applied to the subjects of this survey was adapted to the study material (video lessons), as shown in Table 1, Chart 1 shows the standard wording of the items, according to the original IMMS, translated and cross-culturally adapted into Brazilian Portuguese11.
Chart 1 Validated Brazilian Version of the Instructional Materials Motivation Survey (IMMM-BRV).
| Validated Brazilian Version of the Instructional Materials Motivation Survey (IMMS-BRV) | Discordo totalmente | Discordo parcialmente | Não concordo, nem discordo | Concordo parcialmente | Concordo totalmente | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1. Há 25 afirmações neste questionário. Por favor, pense sobre cada sentença em relação a cada material didático que você acabou de estudar e indique o quão verdadeira é a sentença. Dê a resposta que verdadeiramente se aplica a você, e não aquela que você gostaria que fosse verdadeira, ou aquilo que você pensa que outros querem ouvir | ||||||||||||
| 2. Pense em cada afirmação isoladamente e indique quão verdadeira ela é. Não se deixe influenciar por suas respostas a outras afirmações. | ||||||||||||
| 3. Registre suas respostas na folha fornecida e siga as instruções adicionais que podem ser fornecidas em relação à folha de resposta que está sendo usada nesta pesquisa. Obrigado. | ||||||||||||
| 1. Este material foi mais difícil de entender do que eu gostaria. | ① | ② | ③ | ④ | ⑤ | |||||||
| 2. Depois de ler as informações iniciais, eu me senti confiante de que sabia o que se esperava que deveria ser aprendido com essa atividade. | ① | ② | ③ | ④ | ⑤ | |||||||
| 3. Completar os exercícios dessa atividade me deu um sentimento satisfatório de realização. | ① | ② | ③ | ④ | ⑤ | |||||||
| 4. Muitas das páginas tinham tanta informação que foi difícil escolher e lembrar os pontos importantes. | ① | ② | ③ | ④ | ⑤ | |||||||
| 5. Esses materiais são interessantes. | ① | ② | ③ | ④ | ⑤ | |||||||
| 6. Havia histórias, figuras ou exemplos que me mostraram como esse material poderia ser importante para algumas pessoas. | ① | ② | ③ | ④ | ⑤ | |||||||
| 7. A qualidade da escrita ajudou a manter minha atenção. | ① | ② | ③ | ④ | ⑤ | |||||||
| 8. Essa atividade é tão abstrata que foi difícil manter minha atenção nela. | ① | ② | ③ | ④ | ⑤ | |||||||
| 9. Enquanto eu trabalhava nessa atividade, estava confiante de que poderia aprender o conteúdo. | ① | ② | ③ | ④ | ⑤ | |||||||
| 10. Gostei tanto dessa atividade que gostaria de saber mais sobre este assunto. | ① | ② | ③ | ④ | ⑤ | |||||||
| 11. As páginas desse material parecem pouco interessantes. | ① | ② | ③ | ④ | ⑤ | |||||||
| 12. A forma como a informação está organizada nas páginas ajudou a manter minha atenção. | ① | ② | ③ | ④ | ⑤ | |||||||
| 13. Esta atividade tem elementos que estimularam minha curiosidade. | ① | ② | ③ | ④ | ⑤ | |||||||
| 14. Gostei muito de estudar essa atividade. | ① | ② | ③ | ④ | ⑤ | |||||||
| 15. A quantidade de repetição nesta atividade me fez ficar, eventualmente, entediado. | ① | ② | ③ | ④ | ⑤ | |||||||
| 16. O conteúdo e o estilo de escrita neste material dão a impressão de que vale a pena conhecê-lo. | ① | ② | ③ | ④ | ⑤ | |||||||
| 17. Aprendi algumas coisas que foram surpreendentes ou inesperadas. | ① | ② | ③ | ④ | ⑤ | |||||||
| 18. Depois de trabalhar nessa atividade por algum tempo, eu estava confiante de que seria capaz de passar em um teste sobre ela. | ① | ② | ③ | ④ | ⑤ | |||||||
| 19. A maneira como foi dado o feedback após os exercícios, ou outros comentários na atividade, ajudou-me a sentir recompensado pelo meu esforço. | ① | ② | ③ | ④ | ⑤ | |||||||
| 20. A variedade de trechos de leitura, exercícios, ilustrações etc., ajudou a manter minha atenção na atividade. | ① | ② | ③ | ④ | ⑤ | |||||||
| 21. O estilo de escrever é entediante. | ① | ② | ③ | ④ | ⑤ | |||||||
| 22. Há tantas palavras em cada página que é irritante. | ① | ② | ③ | ④ | ⑤ | |||||||
| 23. Eu realmente não consegui entender muito o material desta atividade. | ① | ② | ③ | ④ | ⑤ | |||||||
| 24. A boa organização do conteúdo me ajudou a ter certeza de que eu aprenderia este material. | ① | ② | ③ | ④ | ⑤ | |||||||
| 25. Foi um prazer trabalhar em uma atividade tão bem planejada. | ① | ② | ③ | ④ | ⑤ | |||||||
| Itens distribuídos por dimensão | ||||||||||||
| Dimensão | Itens | |||||||||||
| Interesse | 5 | 6 | 7 | 10 | 11* | 12 | 13 | 16 | 17 | 19 | 20 | 25 |
| Confiança | 2 | 3 | 9 | 14 | 18 | 24 | ||||||
| Atenção | 4* | 15* | 21* | 22* | ||||||||
| Expectativa | 1* | 8* | 23* | |||||||||
* Inverted items: the score of the items must be inverted.
DISCUSSION
The importance of instruments that measure student motivation in different teaching-learning environments, with adequate evidence of validity and reliability, initially motivated the translation and cross-cultural adaptation of the IMMS into Brazilian Portuguese11. The translation and cross-cultural adaptation of instruments based on solid constructs, created in other languages, saves time and resources19),(20.
In this sense, the present study sought evidence of construct validity of the translated version of the IMMS into Brazilian Portuguese, through the analysis of its internal structure, against a pedagogical strategy that became mandatory during the social isolation period due to the COVID-19 pandemic. which it seems, will be increasingly used even after the need for distancing is overcome: the asynchronous video lessons in the context of remote or hybrid teaching21)-(23.
The necessary premises to demonstrate the convenience of the PCA were adequately verified by the sample adequacy measure (KMO) and Bartlett’s test of sphericity. After the exclusion of 11 items, the IMMS-BRV instrument now has 25 items divided into 4 dimensions. This behavior was also observed in other studies that empirically assessed the validity of the IMMS, with the exclusion of 16, 24 and 17 items having been reported in the three published studies12)-(14.
It is observed, as described in the results, that the PCA grouped the items in the IMMS-BRV in two dimensions close to the original IMMS, maintained as Confidence and Attention. However, two new dimensions (Interest and Expectancy) were created according to the content analysis and with the higher loads of the items grouped in them. In this regard, it is worth noting that the ARCS model, from which the IMMS derives, is based on the expectancy-value theory, whose construct focuses on the interest and expectancy for learning4),(5.
Another relevant fact is that the IMMS was originally validated using undergraduate students and self-directed instructional material printed on paper, quite different from the material used in this and other studies that analyzed its validity15. Furthermore, there was no empirical data treatment for the distribution of items in the four original dimensions, as in the present study. It is possible that such procedure, had it been carried out, would have generated different aggregation of its items in relation to the dimensions described in the original IMMS.
The PCA, conducted in the present study, showed significant loads (0.529 to 0.790), which characterized the adequate correlation of each variable (item) with the corresponding dimension (construct validity)17. Regarding the adequacy of content analysis, in redefining the nomenclature of the dimensions, although there are differences between the IMMS-BRV and the IMMS, regarding the distribution of items and the designation of components (dimensions), the coherence with the constructs that support the original IMMS was maintained3),(24.
These dimensions relate the IMMS-BRV construct to the one of the first motivational design model proposed by Keller3 and described in the introduction. In it, the four dimensions are: Interest, Expectancy, Relevance and Result. Therefore, the IMMS-BRV has two dimensions of the first motivational design model (Interest and Expectancy) and two dimensions of the second motivational design model (Attention and Confidence), showing how the concepts underlying it are in line with the evolution of the motivational design construct postulated by Keller3),(4),(15. Given this composition of dimensions, the issue is highlighted that, if the initial pool of items, from which the IMMS was derived, had been submitted to multivariate techniques to define its dimensionality, the result would not have been closer to that found in the present study and in the others that evaluated the internal structure of the original instrument. Therefore, the results of the PCA and the adequacy of the content analysis in redefining the dimensions indicate initial evidence of construct validity and the Cronbach’s alpha coefficients, found for the dimensions, disclose the good reliability of the IMMS-BRV.
Correlating with the literature, the studies that analyzed the evidence of the IMMS validity also resulted in a reduction in the number of items and the modification of their distribution in dimensions, once again highlighting the situational characteristic of the instrument and emphasizing the need to modify it to accommodate the context of its application12)-(14. In other words, it is suggested that the situation of the instrument is a dynamic one and varies according to factors correlated to its implementation, which range from the students’ characteristics to those of the involved instructional programs.
In this sense, Huang et al.12 evaluated the validity of the IMMS in first-year engineering students who used a computer-based tutorial as supplementary material for learning the use of a computational tool called MATLAB. The result was the maintenance of 20 items distributed in four dimensions, differently from the original instrument, as it occurred in the present study. Also correlating with the present study, Attention was the dimension that maintained the highest number of items in both of them and, of the 20 items maintained in the instrument validated by Huang, 15 (75%) items were also maintained in the IMMS-BRV, representing 62.5% of IMMS-BRV.
Regarding the study by Huang et al.12, Keller4 added that the factorial analysis, when applied to the IMMS, can change its structure, because the dimensions can show very high intercorrelations. He explains that this occurs partly because the instrument was designed to measure situation-specific attitudes rather than psychological constructs. In fact, in the present study, the correlation between the Interest dimension and the Confidence dimension was also high (r = 0.81). In the ARCS construct, the student’s interest in the instructional activity is directly related to their confidence that they will be successful in the task, that is, in learning. Therefore, the items in these dimensions measure behaviors that walk in parallel, because the represented latent traits are highly intercorrelated. However, for the purpose of diagnosis and correction or improvement of the instructional activity being evaluated, it is important to maintain these dimensions separately.
In turn, in the study by Loorbach et al.13, carried out in a self-directed learning environment, aiming at teaching tasks related to the use of cell phones by the elderly, the instrument was reduced to 12 items that adequately fit in the four original dimensions of the IMMS, with the proposal of the instrument called the “Reduced Instructional Materials Motivation Survey (RIMMS)”. Of the 12 items present in the RIMMS, 10 (83.3%) items are also allocated in the IMMS-BRV.
Recently, Hauze et al.14 validated the IMMS in a simulated training scenario applied to Nursing students. In this study, 19 items were maintained, distributed in 4 dimensions, according to the original IMMS.
Based on the analysis of these studies that have been published so far, a lack of homogeneity was observed in relation to the results of the IMMS validation, when applied to different contexts, related to the characteristics of the students and the evaluated instructional material. Thus, in the present study, motivation was analyzed in a different context than those previously researched in studies evaluating the validity of the original instrument. The final version of the IMMS-BRV, reduced to 25 items, makes its application more convenient in the evaluation of remote teaching strategies, such as asynchronous video lessons, as it can measure motivation while requiring less time to fill it out and, consequently, resulting in less fatigue and dispersion of attention when interpretating the different items, therefore, in line with the principle of parsimony of measuring instruments24.
The reliability of the IMMS-BRV, measured by Cronbach’s alpha, ranged from 0.76 (Attention) to 0.93 (Interest) in its dimensions, thus showing good internal consistency25. These findings, compared with other validation studies, show that the items of the IMMS-BRV demonstrate reliability, to the same or to a better degree than those verified in such studies12)-(14. In addition, the IMMS-BRV maintained 80% of the reverse items contained in the original instrument. A similar finding was verified by Hauze et al.14, demonstrating the resilience of the IMMS questionnaire to response bias and to the respondents’ fatigue.
This study has limitations that must be considered, especially in relation to the generalization of results. This is a study carried out in a single medical course, using a convenience sample from a very specific population. Moreover, the instructional activity assessed by the instrument was also unique, as it only included video lessons on the human anatomy strategy. Added to this fact is the atypical period of the Covid-19 pandemic, during which data were collected, as well as the online application of the instrument. The way the data analysis of the variables generated by ordinal measures converted into numbers was performed, as in the case of those used in Likert scales, also raises debate in the literature, since the differences between the answers are not equidistant26.
Finally, it should be noted that this study was aimed in evaluating the validity of the IMMS in the remote learning scenario (asynchronous video lessons). Its results and data available in the literature emphasize the situational characteristic, as well as the non-universal nature of the IMMS and reinforce that the validity and reliability of the IMMS-BRV, demonstrated in this study, should be considered preliminary. Thus, potential uses in other contexts must be evaluated and further studies that seek to corroborate the IMMS-BRV validity in other teaching-learning contexts, giving it external validity, must be carried out to expand its application spectrum. Therefore, the accuracy of the generalization of its use, in contexts different from the one used in this study, remains unknown.
CONCLUSIONS
The application of the IMMS in the remote learning scenario, through asynchronous video lessons on human anatomy, resulted in an alternative instrument (IMMS-BRV), modified with a smaller number of items (more parsimonious) and good internal consistency, demonstrating preliminary evidence of adequate validity and reliability in the employed context.
