Introduction

The research hereby disclosed was carried out in the capital of a Brazilian state. In this city, by decision of the Municipal Government, in-person classes were suspended in mid-March 2020, in all educational institutions. This measure complied with the World Health Organization sanitary protocols to reduce the spread of the COVID-19 pandemic.

The subjects of our research are students from a high school that provides professional education. When this text was written, in-person classes remained suspended. To designate this reality, we make use of the expression Emergency Remote Teaching (ERT) instead of Distance Education (DE), as proposed by ^{Arruda (2020}), as well as by ^{Saraiva, Traversini and Lockmann (2020}).

On the one hand, Arruda (idem) differentiates DE from ERT in order to: (1) criticize what he considers to be a prejudiced idea about DE, which stems from the misconception that this teaching modality is inferior to traditional education; (2) highlight the precarious quality of many ERT experiences, which differ from DE due to the absence of the required infrastructure or subjects with proper training.

On the other hand, Saraiva, Traversini and Lockmann (idem) differentiate DE from ERT based on the different rationalities in which these teaching methods/modalities are grounded. The former is based on a goal-oriented rationality, with no regulation of time or space where students execute tasks. The latter inherited the disciplinary rationality, which is characteristic of traditional education and, this way, transfers the responsibility from the teachers to the families, who are now obliged to exert disciplinary control over the students’ behavior.

In our framework, ERT started in the school in August of 2020, five months after classes had been suspended. Some public school students were unable to access the online environment due to social inequality and the absence of public policies regarding digital inclusion. The school under our investigation assisted the students who could not pay for Internet service. Nevertheless, part of the students accessed the online activities from smartphones. Subsequently, the school rented laptops and lent them to those students.

An almost entirely new course had to be conceived due to the establishment of ERT. The school’s Physics teachers intended to evoke, evaluate, and sustain the students’ engagement. Such fact could be acknowledged because one of the authors of this paper is part of this group of teachers. These teachers prepared a self-administered and anonymous questionnaire in which students were invited to evaluate the first module of the course, in addition to their own engagement in the ERT teaching and learning activities.

Our research group decided to analyze the data collected from the questionnaire in order to answer the following questions: (a) was there student engagement with the course? (b) if so, how to characterize this engagement?

Besides this brief introductory section, this paper consists of four other sections. In Section II, there is a description of the characteristics of the school and the Physics education provided to students before and during the pandemic; in Section III, we present the research theoretical framework and methodology; in Section IV, there is a description of the processes behind the construction of our data; in Section V, we present a general discussion of the results and research limitations, as well as the possible contributions to other teachers and researchers.

Research context

This research took place in a federal public school. Students join the school through a selection process.¹⁴ The experience of two of the co-authors of this paper in class councils shows that teachers’ assessment of the students’ involvement in different subjects is considerably varied. The performance of the students on the proposed activities, as well as their grades, also varies significantly depending on the subject. These findings indicate that student engagement in the school depends on the school subject.

In the Physics subject, most students attain satisfactory levels of academic performance, even though many complex topics are included in the course.² We have carried out studies in which students were asked to evaluate the first grade of the introductory Physics course in the school (^{Paula and Talim, 2012}, 2015). The strategy of having students rate the Physics course is a recurring practice in the school. Successive ratings indicate that most students are interested in the course topics.

This history with several signs of student engagement in the Physics course did not give the team of teachers good expectations in relation to the ERE, because the course underwent some profound changes. In our subject school, students have four fifty-minute Physics classes a week, grouped in sets of two classes. In one of the classes from the set, all students are gathered in a conventional classroom. In the other, students are gathered in a lab where they form small groups. These groups sit at rectangular tables.

In both settings, students often gather in groups and handle experimental materials. All the concepts, models and theories addressed in the course are presented as resources to the interpretation of experimental results, technological devices or phenomena present in their daily lives. Mathematical formalism is employed insofar as it contributes to these interpretations. Furthermore, there is a great rhetorical investment in the proposal and justification of problems on which each activity is based. This orientation is coherent with the epistemology developed by ^{Bachelard (2001}, p. 166), who states that all knowledge is the answer to a question/problem, and that if there is no problem, there cannot be scientific knowledge.

In the course, there was a great deal of interaction between the students and the Physics teacher, and also between the students and their peers. The interpretation of the phenomena discussed throughout the course is done in a collaborative manner, through discursive interaction between teacher and students, as well as between students and their peers. Activities assigned to be done at home are rare and structured so that they may be done in little time. Hence, it is fair to say that the course is almost completely face-to-face and predominantly experimental.

Because of ERT, activities that had been refined over the years proved to be useless. Digital apps with simulations of physical phenomena, which had previously been auxiliary resources in the “real lab”, turned into the predominant mediational resources of the proposed activities. Furthermore, ERT eliminated the possibility of a synchronous space where meaning could be negotiated through interaction between students and teachers, because the pandemic and the available conditions for the continuity of the pedagogic work demanded a predominantly asynchronous setting.³

This kind of ERT hampered the hearing and reception of the students’ contributions to the interpretation of real events and phenomena (^{Freitas e Aguiar Júnior, 2012}), as well as the use of questions to guide them through the scientific concepts, models, and theories (^{Chin, 2007}). Since students spend half of the course load inside the labs where they worked in small groups, peer interaction - whose importance to learning has already been discussed by many researchers (Driver et al., 1999; ^{Barbosa e Jófili, 2004}) - was also restricted.

When ERT started, some teachers and school staff consulted all the teachers and decided to divide the school subjects into two groups. The subjects were organized in modules. For most teachers, there is now an interval of at least four weeks between two consecutive modules of interaction with students.

Before the pandemic, the school had already made an arrangement with Google and started to use the GSuite platform.⁴ At the beginning of the pandemic, the student council did a survey among the students and claimed for the utilization of a single platform for the Emergency Remote Teaching. Students predicted that they would have difficulty learning how to use different platforms. The school’s previous agreement with Google imposed an intensive use of the GSuite apps.⁵

Subject to this reality, the Physics teachers had to conceive a course fully held on this platform. For each topic of the Physics ERT, an assignment was designed on Google Forms and a chat room was created on Google Chat. The chat rooms associated with each assignment were the main channel of asynchronous interaction between teachers and students. Synchronous interaction was restricted to weekly online meetings through Google Meet. Attendance in these meetings or watching their recordings, however, was not mandatory. The attendance rate in the synchronous meetings was low. For this reason, we believe that the students’ course evaluation - which formed the research data - was focused on the asynchronous activities.

As far as possible, the online activities that were structured as forms followed some of the basic guiding principles from the Physics pedagogical project. The progressive introduction of Physics concepts was done through videos that presented either real experiments or explorations in virtual labs. In this manner, the concepts continued being used as resources to the interpretation of experimental results and daily situations. Furthermore, students continued being requested to make predictions, formulate hypotheses, and articulate explanations to physical phenomena to which they are exposed.

Theoretical framework and research methodology

All Physics teachers in the school hold a PhD in Science Education and have research experience. One of the first grade teachers, who did not participate in this research, investigated student engagement in his former research and used ^{Fredricks et al. (2004}) as one of his main references.⁶ This is relevant because it allows us to understand the choice of this team of teachers towards developing a questionnaire to evaluate student engagement in the course.

The research group is composed of one of the first grade Physics teachers, a retired teacher from the same school, and two undergraduate students. The research group proposed modifications on the first version of the questionnaire made by the first grade Physics teachers. The questionnaire is, therefore, a result of cooperation between the research group and the teacher team.

The data analysis and discussion happened on online meetings with the members of the research group. The undergraduate students attended all meetings and were partially responsible for the literature review. Such review could not be presented in this article because the text reached the maximum size allowed by this journal and we preferred to favor the exposition of the data and the details of our analysis procedures in the available space. The second co-author conceived the architecture of the statistical analysis. The first coauthor advised the undergraduate research students, was partially responsible for the questionnaire since its first version,⁷ and was the main writer of this paper.

The fact that the data were collected from a questionnaire conceived and used by teachers as part of their teaching strategy led us to believe that submitting this research to an ethics committee was not necessary, although this would have been ideal. Since we committed to present the data and analysis to the teacher team in order to make eventual changes in ERT that could evoke or sustain student engagement, our research schedule was tight. Because of this agreement between the teacher team and the research group, the research had to be done within a timespan of four weeks.

We decided to carry out the research without formal consent of an ethics committee because we took into consideration the fact that our data were not collected solely for the research, and that the questionnaire was optional and anonymous. Furthermore, the teacher team authorized us to do the research.

Student engagement with the school and with specific school subjects became an eminent object of research throughout the 1990s and the beginning of the 2000s. In spite of the great variety of conceptions of engagement (^{Christenson, Reschly e Wylie, 2012}), there is a certain consensus that it is a predictor of school achievement. Similarly, disengagement is considered to be a predictor of school dropout and it is associated with a series of behaviors and attitudes that precedes it, such as poor attendance, poor performance in tests and exams, apathy, and disruptive behavior (^{Klem & Connell, 2004}).

Another general agreement found in the literature (^{Christenson, Reschly e Wylie, 2012}; ^{Fredricks et al., 2004}; Fredricks et al., 2016) is the affirmation that the concept of engagement is multidimensional and that it involves several factors (emotional, cognitive, behavioral, attitudinal, social, among others), though they can vary in number and in the indicators of each dimension. In our research, we adopted the concept of engagement developed by Fredricks et al. (2016), for whom it involves four irreducible, interdependent and inseparable dimensions: behavioral, cognitive, emotional and social. It is important to clarify that, although the concept of engagement involves several dimensions - according to different authors - it should be interpreted as a single construct.

According to ^{Fredricks et al. (2004}), behavioral engagement refers to students’ compliance with school rules and absence of disruptive behavior, as well as participation, attention, persistence, and effort in the school activities. Emotional engagement concerns the affective reactions of students to teachers and activities, as well as the feeling of belonging and identification with the school. Cognitive engagement is related to students’ investment in learning, which involves metacognitive strategies to solve problems and understand complex ideas. The social dimension of engagement, as described by Fredricks et al. (2016), refers to the quality of the social interactions that are part of the learning and teaching environment.

Considering our collaboration with the group of teachers that created the questionnaire from which we collected our data, we suggested the use of the concept of engagement described by ^{Fredricks et al. (2016}). Besides these authors’ work, we found in the literature several other instruments for measuring engagement via self-administered questionnaires (^{Wang et al., 2016}; ^{Gale et al., 2015}; ^{Wang et al., 2014}; ^{Wang et al., 2019}; ^{Veiga, 2016}). These instruments, however, were designed to be used in face-to-face courses. We found a small number of works that involved Distance Education (^{Nasir et al., 2020}; ^{Dixson, 2012} and ²⁰¹⁵).

Many of these authors carried out pilot studies with samples of students as a validation step to their research instrument. Subsequently, these researchers interviewed the students in focal groups in order to understand their interpretation of the questionnaire content. We did not apply these validation steps to our instrument. Firstly, because it was not conceived solely for our research. Secondly, because we did not have the time or the necessary conditions to do so. For these reasons, we did not completely validate our research instrument, although we did carry out satisfactory procedures that allowed us to answer our research questions, as described in the next section of this paper.

Our questionnaire was inserted on a form entitled “Evaluation of the Module about Waves”. In its introduction, all the three Physics teachers who were responsible for the course confessed the challenge of transforming a course that was previously 100% face-to-face and predominantly experimental into a course with asynchronous and online activities. In the same introduction, they also committed to improve students’ experiences during the next ERT module. Finally, the introduction also informed students that all collected data would be anonymous and that answering the questionnaire was optional, although of utmost importance to help enhance the course.

Following the introduction, the form contained twenty-seven statements, which are listed in the Appendix. They were divided into four sections themed as: Behavior and Attitude; Emotions and Feelings; Learning Strategies; Interactions with the Teacher and Classmates. Students had to interpret the statements and mark one of the numeric options of the following Likert scale: 1 - Strongly disagree; 2 - Partially disagree; 3 - Neither disagree nor agree; 4 - Partially agree; 5 - Strongly agree.

An exploratory factor analysis was performed on the collected data in order to verify whether the four dimensions of engagement (behavioral, emotional, cognitive and social) were supported by the factor structure. We also evaluated the reliability of the total engagement scale and the individual scales for each identified factor. The results of the analysis, performed by the software SPSS⁸ (version 23) and JASP⁹ (version 0.13.1.0), are described in the following section.

The last section of the form was entitled “Open field for suggestions and comments” and requested students to help the teachers improve the ERT imposed by the COVID-19 pandemic. This was the only space for a discursive answer on the form.

Construction, presentation, analysis and interpretation of data

Out of the 172 students regularly enrolled in six classes, only 99 answered the questionnaire, which corresponds to about 58% of the students. In order to understand this rate of adherence to the evaluation invitation for the first module, it is important to consider that: 1st - the students had already done twelve ERT Physics activities; 2nd- both remote activities and research questionnaire were presented to students through forms. Thus, the rate of adherence to the questionnaire may have been limited by: (a) the tiredness of the students, who were demanded to answer successive forms; (b) the fact that the answer to the survey questionnaire is presented as anonymous and optional; (c) the fact that the students had already started studying a new group of subjects with which they were beginning to interact, for the first time, in the same period in which they were invited to answer the questionnaire.

3. Data and research questions

At this point of the analysis, we can go back to our research question. The data presented in the previous subsections are evidence of the reliability of the scales that we created to assess student engagement in the course, and also helped us qualify this engagement.

Considering the data in Table 2, we see that the mean values for Cognitive-behavioral, Emotional and Social engagement are different. Before interpreting these differences, we decided to check whether they were statistically significant. To achieve this, we performed the ANOVA test of repeated measures in SPSS, as well as a post hoc Bonferroni test, in the same program. Combined, these tests showed that all differences were significant [F(1.72; 168.63) = 17.57; p < 0.001]. Table 2 shows that Cognitive-behavioral engagement was the highest, followed by Emotional and Social.

The mean values of the answers given to each of the twenty-five statements collected by our instrument allow us to better understand how student engagement varied. If we consider that values greater or equal to 4.0 indicate high engagement, we see that there are: (a) three statements of high engagement (Q_17, Q_18 and Q_21) in the Cognitive-behavioral dimension (43% of the items);¹⁰ (b) two statements of high engagement (Q_08 and Q_20) in the Emotional dimension (22% of the items); (c) no statement of high engagement in the Social dimension.

The three statements of high engagement in the Cognitive-Behavioral dimension indicate that the majority of students: (a) tried to relate what they were studying with what they had learned from previous remote activities. (b) tried to understand their mistakes when they realized that they had made them; (c) strived to perform tasks and learn, even when they had difficulty. These statements are clear examples of what ^{Fredricks et al. (2004}) consider to be expressions of cognitive engagement and, more specifically, of metacognitive strategies to understand complex ideas. The two statements of high engagement in the Emotional dimension, in turn, indicate that most of the students: (a) were interested in the subjects; (b) did not give up on dealing with subjects that are difficult to understand or abandon them to perform other tasks.

We can interpret these findings as an indication that the activities of the Physics ERT allowed students to: (a) maintain their interest; (b) overcome the obstacles that they were faced with; (c) establish relationships between the studied phenomena and the concepts introduced to understand them.

A second way of interpreting the same findings, which does not contradict the previous one, concerns the students’ characteristics. In the section devoted to describe the research context of this manuscript, we drew attention to the fact that the students join the school through a selection process. This type of process tends to select students who have managed to remain engaged in their previous school experiences and who, for this reason, probably have the necessary conditions to learn Physics in the unfavorable conditions of the ERT provided by the school where we conducted the research.

Once we have addressed, in the previous paragraphs, our interpretation of the evidence of cognitive-behavioral and emotional engagement, we decided to end this section focusing on the evidence of low student engagement in the Social dimension. Only one statement associated with this dimension (Q_27) had a value significantly greater than three (average value of 3.84 in the Appendix). This is the statement in which students say they have used “other communication resources (WhatsApp, for example)” to interact with their peers. In a complementary way, statements Q_05 and Q_06, which have the two lowest averages among all items in the questionnaire, are precisely those that express teachers’ insistence on the use of chat rooms as an asynchronous interaction channel.

Data obtained in the discursive question of the questionnaire - which have not yet been mentioned in the analysis so far carried out - help us complement our interpretation of student engagement in the course. Thirty-three of the ninety-nine students who responded to the form used the “Open space for suggestions and comments”. We carefully read the responses whose content could be summarized in the following categories: positive and negative highlights of the course, suggestions, complaints and compliments. We shall focus on the most frequent instances.

Ten out of the thirty-three responses referred to the use of explanatory videos about the contents of the activities. The answers varied among: (a) compliments about the introduction of these resources from the middle of Module I on, as a result of a claim made by many students; (b) complaints about the absence of this resource at the beginning of the Module. Seven students used different words and language resources to ask teachers to “teach the subject” before asking questions. Ten other discursive responses presented suggestions or complaints on how teachers chose to interact with the classes: Google Chat. Six students addressed this issue by directly criticizing the mandatory use of the Chat. Four other students requested the use of WhatsApp instead of Chat.

In the presentation text of the second ERT Physics module, the teachers responded to some of the students’ requests in the open response questionnaire item. The teachers committed to adopt some of the suggestions and presented justifications for declining two specific requests. The first request refused was that to “explain the subjects before asking questions.”

As a justification for this refusal, teachers said it was important for students to understand how to answer questions about subjects that they do not fully comprehend yet because: (a) dealing with the discomfort provoked by this type of situation is an important part of the scientific learning process; (b) in order to study a science, it is necessary to learn how that science works and how scientists produce knowledge; (c) it is very common that, when studying science, there is a need to live with only partial answers to questions and to have to reason from questions, rather than from answers and reliable information. Although it is not our objective, in this paper, to analyze this pedagogical discourse of teachers, it is important to emphasize that it reveals important aspects of the conception of science teaching and learning in this group of teachers.

The second refusal from the teachers concerned the suggestion of using WhatsApp instead of Chat. The teachers stated that they had removed the mandatory use of Chat, but reiterated that the creation of a Chat room for each activity was the best way to allow students to count on their help in case of questions. To justify, they said that: (a) on Chat, it is possible to start a new conversation for each new question; (b) those who access the Chat and find conversations that have already been started can insert their question or comment in these conversations or start a new conversation with a different question; (c) in Chat it is easier to find answers to questions already addressed by the teacher; (d) none of these features exist on WhatsApp, where it is difficult to locate and resume an issue previously discussed.

Conclusion

The research hereby disclosed presented evidence of the engagement of students who attended an introductory, remote and emergency Physics course in adverse conditions. The results of the survey help to understand which aspects of the course elicited greater and lesser engagement.

Instead of the four dimensions of the engagement construct validated in the work of ^{Fredricks et al. (2016}), our exploratory factor analysis showed the behavioral and cognitive dimensions merged into a single factor.

This result can be interpreted not only as an effect of our sample size, but also as a result of the following limitations of our research: (a) we did not involve independent experts in the development of our construct;¹¹ (b) we had a short time to perform the research and this made it impossible to carry out qualitative pilot studies with focus groups and/or interviews to assess how students interpreted the questions or to request explanations for the answers that these subjects gave to the instrument; (c) we did not triangulate our data with other sources of information (recordings of meetings held on Meet or analysis of interactions on Chats created for each activity, for example).

Being fully aware of those limitations, we consider that it is important to state that our research instrument: (a) was not submitted to a strong and complete validation; (b) cannot be used to measure student engagement in remote classes in other contexts. Even with those limitations, our research built evidence of student engagement and allowed us to qualify that engagement.

In addition, we would like to say that we work in cooperation with the team of teachers responsible for the course and that part of our data has had a significant and immediate impact on improvements in the following modules of the course. Accordingly, the main merit of our research is pedagogical and not theoretical or methodological. It is difficult for teachers to carry out research with short deadlines and in time for their results to interfere with planning and pedagogical practice. Hence the importance of collaboration between teams of teachers and researchers, whenever possible.

A featured characteristic of several engagement measurement or assessment instruments that we found was the adjustment of those instruments to the specificities of academic activities to which the students would or would not be engaged. This is a characteristic that hinders a complete and unadapt reuse of previously validated instruments in new contexts.

The evidence we found came from statistical analysis procedures accessed at two complementary programs: SPSS and JASP. The procedures used were presented and described at the fourth section of this paper. Our results indicate that a significant amount of students present engagement at essential aspects of learning associated with the Emotional and Cognitive-behavioral dimensions.

That is a remarkable result when considering the adverse conditions in which the ERT was instituted at the school.¹² Regarding it, in the introduction of this paper we draw attention to the fact that several students accessed the proposed activities from their cellphones. On the other hand, in the section dedicated to present the theoretical framework and describe the methodology, we said that teachers had acknowledged their difficulty for turning a 100% face-to-face and strongly experimental teaching approach into a course mediated by asynchronous and remote activities.

As ^{Dixson (2015}), we believe that the engagement evaluation of distance or online courses can guide teachers to promote the required adaptations aiming to raise or increase the students’ involvement with their own educational experiences. Unlike this author, though, we believe that face-to-face courses have more and better resources than distance courses. Particularly, in our case, we consider that the context experienced by the students at the ERT were exceptionally unfavorable.

Thus, in spite of the enormous pedagogical effort undertaken by the teachers: (a) there was a sudden transformation from a 100% classroom-based and predominantly experimental course to a remote course; (b) laboratory classes with real equipment have become unviable, which has resulted in an impoverishment of the mediational resources used in the course; (c) the interpretation of phenomena and the use of ideas from the sciences could not continue to be carried out collaboratively through synchronous discursive interactions between the teacher and the students, and the students and their classmates; (d) there was a drastic reduction in the possibilities of students’ social interaction with their teachers and peers.

Thinking about the relevance of the work that we present here to the research community, we believe that one of its positive points was the consequent use of the possibilities offered by statistics for the understanding of educational phenomena, even in situations where it is not possible to carry out a methodologically robust research. Finally, we also believe that our research contributes to a new field of research dedicated to study the impacts of the COVID-19 pandemic on school education.

Considering the fact that we are still amidst a pandemic and the production and publication of research requires time help us understand the limitations of the quantity and quality of the works available on the topic when our paper was finalized. Some articles are experience reports (^{Barbosa, Ferreira and Kato, 2020}; ^{Hoffmann et al., 2020}; ^{Silva and Ramos, 2020}; ^{Piffero et al., 2020}), others have methodological flaws that compromise the work (^{Médici et al., 2020}). There are also those who present the imposition of remote education as an opportunity for “innovation towards education” without, however, displaying consistent data to corroborate this claim (^{Carneiro et al., 2020}; ^{Bezerra et al., 2020}).