Introduction

In March 2020, Brazilian schools interrupted face-to-face teaching due to the COVID-19 pandemic. At the time, health authorities believed that the subjects of educational institutions would be major agents of transmission of the virus. Although students were not among the population at greater risk and, in most cases, were asymptomatic, in commuting to school, they could involuntarily spread the virus. Thus, it was believed that closing schools and adopting social distancing would slow down the spread of the virus.

Schools interrupted face-to-face classes, reorganizing schoolwork, opting for the continuity of the school year through distance, non-face-to-face activities (^{CUNHA; SILVA; SILVA, 2020}), among which some institutions have adopted Emergency Remote Teaching (ERT), mediated by digital information and communication technologies. It is important to highlight the difference between teaching experiences specifically designed for the virtual environment and online courses offered in response to crisis situations. The term “remote teaching” refers to the physical distance between teachers and students, the main actors in the educational process (^{DIAS-TRINDADE et al., 2020}). The term “emergency”, on the other hand, means that the change from face-to-face to remote teaching occurred without enough time for effective pedagogical changes related to the new teaching environment (DIAS-TRINDADE et al., 2020). As stated by ^{Hodges (2020}), the expression emergency remote teaching emerged as an alternative to highlight the difference between online teaching offered in unexpected circumstances that prevent face-to-face teaching, as in the case of the pandemic, and Distance Education, which results from instructional planning and design originally thought for the virtual environment. For this author, ERT, “in contrast to experiences that are planned from the beginning and designed to be online, emergency remote teaching (ERT) is a temporary shift of instructional delivery to an alternate delivery mode due to crisis circumstances” (^{HODGES, 2020}, n.p.). Due to the long duration of the pandemic, emergency remote teaching has become the main alternative for educational institutions unable to offer face-to-face teaching.

The “new normal” of emergency remote teaching required teachers, technical staff, students and family members to have access the internet and digital technologies skills, which play a fundamental role in promoting access to teaching and learning processes.

However, the abrupt shift from face-to-face to online presented challenges for the school community. Equity in relation to access to digital technological resources is one of these challenges, especially in a country with a continental extension like Brazil. Not all students have access to Internet or devices such as smartphones, tablets and computers. For example, in the Southeast region, where this research was carried out, 25% of households do not have access to the internet and 54% do not have a desktop computer, a notebook or a tablet (CGI.BR, 2020).

Another challenge is the learning curve necessary to develop skills to use devices and applications aimed at education. Although students are in constant contact with the virtual world, they are often just passive consumers of digital technology and the internet, using them “in the same way as many of their elders: to passively soak up information” (NATURE, 2017, n.p.) and, often, do not have the skills to use educational applications. According to ^{Silveira (2018}), at home and at school, digital practices for study are still scarce, and digital leisure practices prevail. For some teachers, the challenge in relation to digital technologies is also great.

At the same time, time management is one of the most worrying issues in remote teaching, perhaps, since face-to-face work and study at school moved to the homes, in “conflict” with domestic tasks, non-school leisure and interaction. On the other hand, teachers' uncertainty regarding learning expectations caused a focus on the method of delivering instruction rather than the learning goals, leading to uncertainty around assessment for both teachers and students (^{WHITTLE et al., 2020}).

Thus, the sudden implementation of Emergency Remote Teaching posed many challenges, and it was not possible to predict its duration or even its future need due to other crises. Digital technologies will probably occupy more space in education, even after the return to face-to-face teaching.

There is an urgent need to establish assessment indicators for Emergency Remote Teaching, especially for online teaching, so as to make it possible to improve it, leading to more effective teaching processes, in the event of new crises that lead to other sudden interruptions of face-to-face classes and a possible increase in ERT situations in the future (^{WHITTLE et al., 2020}).

The present research aimed at developing and validating an instrument (a questionnaire) to formally assess the organization and development of Emergency Remote Teaching, from the school community’s point of view (teachers, administrative workers, students, and family members) in a Brazilian public school. It is, otherwise, important to emphasize that this study does not intend to assess the learning results of ERT in any of the school curriculum subjects (science, math, language, art, physical education, history etc.).

Emergency Remote Teaching and digital technologies

Covid-19 pandemic brought to light the demand for new and old reflections on teaching and learning processes (^{MARTINS, 2020}). Presently, the relationship between digital technologies and education has been intensified and gained a new meaning (^{DIAS-TRINDADE et at., 2020}).

In the exceptional circumstances arising from the social distancing caused by the covid-19 pandemic, digital technologies allowed millions of students to continue their educational processes, even when restricted to their homes (^{SANZ, GONZÁLLEZ, CAPILLA, 2020}).

Thus, digital technologies have suddenly become the means for the continuity of the teaching process, without enough time for due reflection on the change from face-to-face to virtual (^{RONDINI; PEDRO; DUARTE, 2020}). Despite the need to reflect on the interaction between digital technologies and education, the pandemic has shown that the readiness of teachers, students and family members for the pedagogical use of digital is still not complete (^{DIAS-TRINDADE et al., 2020}).

Even though the introduction of information and communication technologies (ICT) in education has been discussed for a long time, the incorporation of ICT in school is still a challenge in the Brazilian reality (^{BRAGA, 2018}). Problems of infrastructure, equity in accessibility to devices and teacher training are important variables that directly interfere with a reflective, intentional and productive use of digital technologies in education (BRAGA, 2018; ^{THADEI, 2018}).

For ^{Tomazinho (2020}), what is going on is pedagogical planning in real time, and never have schools demanded managers and teachers to make such quick changes. However, research indicates that, despite being challenging (^{AVELINO; MENDES, 2020}; ^{BARRETO; ROCHA, 2020}; ^{MARTINS, 2020}), the school remote experience during the covid-19 pandemic should provoke transformations in the post-pandemic period, in view of the intensification of the use of ICT in education by teachers, students and family members.

^{Barreto and Rocha (2020}) highlight how much teachers have reinvented themselves, in a search for qualification to take the best of ICT in the teaching process. However, the existence of so many different digital technologies make it difficult to all of a sudden choose which ones are most suitable for each educational situation (^{MORAN et al., 2015}). For ^{Netto, Almeida and Souza (2020}), an instrumental appropriation of digital technologies by teachers, to meet emergency demands, is prevailing.

Research carried out by ^{Leite, Lima and Carvalho (2020}) showed that most of the surveyed teachers were not offered qualification to carry out remote activities, having sought knowledge on their own. The research results also highlighted the importance of continuing education and the unpreparedness of teaching systems to provide work alternatives and the necessary theoretical support, forcing teachers to qualify in real time. The difficulty in mastering digital skills and the need to invest in teacher training for the reflective use of digital technologies had already been observed in research prior to the pandemic, as Leite, Lima and Carvalho (2020) point out.

Research indicates that teachers intend to continue to use digital technologies in education (^{RONDINI; PEDRO; DUARTE, 2020}; ^{NETTO, ALMEIDA; SOUZA, 2020}). However, ^{Rondini et al (2020}) observed that teachers who already used some technological resource before the covid-19 pandemic were more likely to continue using it in the post-pandemic period. Thus, the experience of emergency remote teaching suggests that it is necessary to trigger “educational processes aimed at improving and developing the professional quality of teachers” (^{MOREIRA; SCHLEMMER, 2020}, p. 28).

On the other hand, the use of digital technologies must go beyond the mere adoption of applications and software, providing engagement, interaction and interactivity between students and learning activities (^{OLIVEIRA; CORRÊA; MORÉS, 2020}), as well as a critical reflection about digital technologies in the contemporary world, that is, the critical dimension of digital literacy (^{BUCKINGHAM; BURN, 2007}).

From this perspective, research aimed at assessing emergency remote teaching are important in understanding how it occurred and to what extent it needs to be improved not only to develop online teaching processes for future emergency situations like the one we are experiencing, but to provide the reflective use of digital technologies in the post-pandemic period.

Emergency Remote Teaching at Centro Pedagógico

The investigated institution is a federal public elementary school, located in the urban region of Belo Horizonte, Brazil. Founded on April 21, 1954, from 1958 onwards it became a School of Application, currently constituting the School of Elementary and Professional Education, a special unit of the Federal University of Minas Gerais (UFMG).

Responsible for nine-year elementary education, the institution adopts a public lottery for the admission of students, as it considers it the most democratic way, avoiding selection mechanisms that favor any social or cultural groups, with a guaranteed 5% of places for students with disabilities.

Of the 440 students, 52% are male and 48% are female, with 40% self-identified as brown, 38% as white, and 11% as black, in addition to 3% as yellow and less than 1% as Indigenous. The income of 8% of student families is up to one minimum wage; 24% between one and three minimum wages; 28% between three and five minimum wages; 20% between five and ten minimum wages and 6% of families have an income above ten minimum wages (^{SALGADO, 2021}).

The school has two computer labs with fifteen computers connected to the internet each. Most classrooms have a data show projector, but no internet access. Despite belonging to a federal university, which provides access to the wi-fi network to its undergraduate and graduate students, because of their age (6 to 15 years), the students of the investigated school still do not have access to the university wireless network.

The school has sixty-four full-time professors, most of which have a PhD degree, with an average weekly workload of twelve class hours (8 clock hours). The other 32 hours of work per week are dedicated to planning classes and correcting activities, school administration and research projects (related, especially, to teacher training, teaching and learning processes etc.).

On March 18, 2020, through Belo Horizonte town hall Decree 17,304, face-to-face classes at schools were suspended. Two months after the suspension of face-to-face classes, and with no prospect of a return, Centro Pedagógico chose to use the Moodle virtual learning environment to provide emergency remote teaching. It is noteworthy that Moodle was little used as a tool to support face-to-face teaching by school teachers, before the pandemic. A commission composed of teachers and administrative workers was responsible for organizing Moodle as a virtual space for teaching and educational experiences.

To ensure quality access to Moodle for all students, following a survey that indicated the need for support for some families, calls for internet and computer assistance were launched. Forty-two families (9.5% of the students at the school) applied and were granted a monthly allowance to pay for internet access in their homes, while one hundred families (22.6%) were granted a laptop loan (^{SALGADO, 2021}). Among school employees, there were no requests for internet assistance, but five teachers and two administrative workers asked and were granted a computer loan for remote work. The absence of a computer or notebook can become an obstacle for students' performance, since some activities may be difficult to undertake on smartphones (^{CUNHA; SILVA; SILVA, 2020}).

In this context, five months after the interruption of face-to-face classes, in August 2020, emergency remote teaching began at school, for all students from first to ninth grade of Elementary School.

Methodology

This qualitative research initially involved the elaboration of indicators, by the three authors, followed by focus groups that analyzed such indicators and provoked their reconstruction. Finally, the reconstructed indicators were submitted in writing to the research subjects, who validated the changes.

The process of developing indicators

As stated by ^{Minayo (2009}), indicators can be understood as quantitative or qualitative parameters that serve as indexes of reality, supporting the process of investigation. Thus, they are empirical correlates of the variables we are trying to measure; concrete expressions of a phenomenon that is not directly observable.

For the creation of indicators for ERT, it is also important to pursue the classic attributes of a good indicator, although, as Torres, Ferreira and Dini (2003, p. 82) point out, it is difficult to gather all the attributes considered “indispensable to a good indicator, such as: credibility, simplicity, spatial disaggregation, reproducibility, comparability, periodicity, accuracy, low cost and sensitivity”.

Regarding ERT assessment, ^{Oliveira et al. (2020}) organized an instrument with six indicator dimensions, using a 4-point Likert scale (1 = never, 2 = rarely, 3 = often and 4 = always): educational and organizational issues; technological and working conditions, social issues, family issues, psychological issues, financial issues.

^{Cunha, Silva and Silva (2020}) present a series of questions that inspire the construction of ERT assessment indicators, among which we can highlight: the situation of students without internet access, nor are able to commute to school to collect study materials; students who, having access to ERT, do not have follow-up or guidance from their families at home; students who, having access to the internet, do not have the devices to access ERT; teachers' difficulties in properly developing the teaching process with available digital technologies; the quality, the right to and equality of access to education for all students at ERT.

^{Hodges et al. (2020}) present an evaluation of ERT for universities that have implemented it, based on four variables: context, interaction, process and product. For them, ERT assessment should be more focused on context, interactions and processes than on products. Regarding contextual aspects (institutional, social, governmental), the authors highlight: the effectiveness of the transition from face-to-face teaching to ERT; the need to assess the school's interaction with students and families and the impact of this interaction on the response to the shift to ERT; the infrastructure to handle ERT; the ability of the school to support students and families to deal with ERT demands (^{HODGES et al., 2020}).

Reflecting on the ERT design, ^{Whittle et al. (2020}) emphasize the importance of interactions within school community, obvious in the face-to-face activities of the school, and a challenge for ERT, since there is evidence that the different interactions (teacher-student, student-student, family-school) seem to influence learning. In this sense, the interactions made possible or hindered by ERT also take a prominent place in the construction of an assessment instrument.

Thus, based on available academic reflections, four categories of ERT evaluation indicators of the investigated school were initially created for the present study: a) access; b) organization of emergency remote teaching; c) quality and scope of teaching/study strategies; d) technical skills and interactions in emergency remote teaching. However, the exchange of ideas with the research subjects, especially the focus groups, led to a change in these categories, as we will see.

Data analysis, produced from transcriptions of audio and video recordings and from observation records in focus group meetings, was essentially descriptive and interpretive.

The focus groups

ERT is still little known and explored academically. Thus, the research sought to investigate in an exploratory way the experience of ERT during the pandemic, using the focus group as a research instrument. This methodological resource, also known as discussion group, can be used at different times of the investigation process. As a technique, it occupies an intermediate position between participant observation and in-depth interviews (^{MORGAN, 1997}). It can also be characterized as a resource to understand the process of construction of perceptions, attitudes and social representations of human groups (^{VEIGA; GODIM, 2001}).

According to ^{Gatti (2005}), focus groups can be used in social research processes or in evaluation processes, especially in impact assessments. The most usual procedure is to hold more than one focus group in the same investigation, seeking to cover various factors that may be intervening in the question to be examined. Focus groups are also in line with the idea of ^{Minayo (2009}) that a “good system of indicators for evaluation must always emerge from the process of dialogue and negotiation between all actors”, which, in the case of schools, are teachers, administrative workers, students and families.

In order to ensure a representative sample of the diversity of the investigated school, all teachers (64), administrative workers (32) and students’ family representatives (9). Fifteen teachers, four administrative technicians and three family representatives responded to the invitation.

Three focus groups were held, the first with six teachers, the second with five teachers and the third with three administrative workers, each of them lasting about 90 minutes. In each focus group, the questionnaire was presented with the indicators proposed by the research authors, and the focus group participants were asked to evaluate how they perceived each question in relation to the following aspects: relevance, clarity and simplicity of the item; relevance of the item to the indicator (category); need to change the item's position in the questionnaire; need to eliminate the item due to redundancy, non-relevance, etc.; need to include new items that were not initially foreseen. In addition, the research subjects were also asked to assess the relevance of each question for each group in the school community (teaching or technical staff, students and family members).

It was not possible to carry out the focus group with students’ family representatives, which is why the questionnaire was sent to them to be commented and then returned to the researchers. Initially, family representatives understood they should answer the presented questionnaire, but, after a further explanation by the researchers, they understood their role as research collaborators.

These research procedures were important for the investigation, since they allowed the participation of representatives of the entire school community, reflection on emergency remote teaching, and the understanding of the investigation importance by research subjects (since it is in the interest of all to assess as well as to develop ERT). The data produced were the transcripts of the group discussions, family representative commentaries and focus groups moderator's reflections and notes.

Description of the questionnaire validation process

Semantic validation is a step after theoretical validation, carried out through focus groups, and seeks to verify whether the items are understandable to respondents, subjects of the population to which the instrument is to be applied, and whether they cover everything that it is important to evaluate.

The validation process aimed to evaluate each assertion of the questionnaire in relation to relevance, clarity, and simplicity of the item; relevance of the item to the indicator (category); need to change the position item; need to eliminate the item due to redundancy, non-relevance, etc.; need to include items that were not foreseen by the researchers.

Initially, the researchers created fifty-eight items to be reviewed through the focus groups. These fifty-eight assertions were divided into four indicators, namely: access; organization of emergency remote teaching; quality and scope of teaching/study strategies; technical skill and interactions in emergency remote teaching. In addition, open-ended questions were included to allow respondents to explore issues that, from their point of view, were not addressed in the questionnaire or that needed explanation.

We present below the main considerations of the research subjects about each indicator category, which lead to the change (rewriting, inclusion, exclusion, change of order or category) of the original questions of the questionnaire initially constructed by the researchers.

Indicator category: “Access”

Regarding the frequency of internet use, it was suggested that, in addition to what was initially proposed by the researchers, an item should be included that assessed the type of data used by the respondent. According to the research subjects, it would be important to qualify the type of internet connection (wi-fi, public, data plan, cable, etc.), as well as the place of use (home, work, school or elsewhere).

Regarding the main reasons for using the internet before ERT, the need to include an item to qualify such use was indicated, in order to elaborate a scale so that the respondents could determine an order of importance. It was also pointed out the relevance of including different alternatives according to the respondent public (teachers; administrative workers; family members and students).

Regarding the equipment predominantly used to access the internet before ERT, the use of images in the questionnaire was suggested, since some people could not identify or differentiate terms such as desktop, notebook, computer, smartphone, cell phone etc.

Regarding the question about Moodle use before ERT, research subjects proposed the inclusion of an item to qualify the use of available tools, the situations and contexts of use as well as the possibilities of platform customization.

As for computer/notebook and smartphone skills, it was proposed a choice of three levels, that is, basic, intermediate and advanced, with distinctions from turning on/off, choosing a browser and sending e-mail, to using social networks, and editing texts, presentations and videos etc.

With regard to equipment necessary for ERT, the need to reformulate the wording was pointed out, explaining or detailing what is meant by infrastructure (internet, computer, smartphone etc.). As for school participation in solving technical difficulties related to the use of Moodle, it was proposed to substitute the word backup for support. Research subjects also proposed that a question intended to rank the top applications of internet use before ERT be added.

In addition to these changes, research participants suggested the inclusion of the following open questions in the access category:

Besides that, research subjects indicated the need to exclude the following two items, due to the possibility of different interpretations of the meaning of the question and the risk of inducing a socially desirable response, which would invalidate the answers:

Indicator category: “Emergency remote teaching organization”

The title of the indicator category “Quality and reach of teaching strategy” was reformulated to “Emergency remote teaching organization”, due to the consideration of some research participants that quality and reach were constructs that would not be the target of evaluation, since they are more related to learning and ERT results.

Research subjects suggested that the question about organization of remote teaching in synchronous and asynchronous activities had its wording changed, to explain that it was desired to understand the respondents' evaluation regarding the organization of ERT in two different moments: synchronous and asynchronous.

Regarding delivery of asynchronous activities by the students within the deadline stipulated by teachers, it was proposed to replace the word 'adequate' by a scale with its qualification and the addition of a new item, qualifying the synchronous and asynchronous times.

In the question about the technical difficulties of carrying out asynchronous activities in Moodle, specifying the possibilities of difficulty were indicated, that is, photo, audio and video posting, PDF downloading, transforming video file into a link etc., as well as the inclusion of the answers “no difficulties” and “others”, so that difficulties not imagined by the researchers could be presented by the families.

In the question about the school contribution to overcoming these difficulties, in addition to the yes and no options, the inclusion of “sometimes” and the possibility of justifying the choice was proposed.

As for the questions related to difficulties in posting, receiving, correcting activities (for teachers), browsing Moodle, carrying out and posting activities, in addition to difficulties in synchronous classes (for students), the need to modify the formulation was indicated, to ask if there were (yes or no) difficulties, with the consequent qualification (always, almost always, sometimes, almost never, never) in case of an affirmative answer.

The research participants also suggested adding the following questions, related to the amount and duration of synchronous activities:

On the other hand, the exclusion of the following two questions whose contents were already covered was proposed, in a more qualified way, elsewhere in the questionnaire:

Indicator category: “Technical skills”

For all questions related to technical skills and knowledge developed (about technology, internet, virtual learning environment navigation, etc.), to the impact of training actions and school support for the development of such skills, and to the adequacy of ERT skills to students, the importance of creating a scale of these skills was pointed out, so that respondents could choose in more detail.

Indicator category: “Interactions in ERT”

In this category, the suggestion for change focused on the specification of the different interaction possibilities (Moodle chat, WhatsApp, social networks, synchronous class chat, through the school's Multiprofessional Department, through the School Family Council, etc.). Research subjects also proposed the inclusion of each of the school administrative departments (and not just the “school” option as a whole) as possibilities for interaction between school and families (Multiprofessional, Educational Support Sector, Board of directors, Pedagogical Coordination, Information Technology Support, ERT Commission, Library etc.).

Open questions

In the original questionnaire as initially proposed by the researchers, there were only two open questions, namely:

Based on the research subjects' considerations, the first open question was transformed into more questions to make it possible to separately assess the limits and challenges of ERT.

The second question was rewritten, replacing the word “amelioration” with “improvement”, and the three following questions were included, in order to allow respondents to present other points of view about ERT not included in the answer possibilities the closed questions of the four indicator categories.

The final ERT assessment instrument

Eight items were excluded from the original questionnaire for not being sufficiently understandable, for inducing socially acceptable responses, or for being unspecific and not providing elements for analysis. Fourteen items underwent changes in writing, in the first review of the focus group, mainly with the aim of expanding response possibilities or for best understanding of the item. Nine items were added, as the focus group participants suggested questions that the initial instrument did not contain.

After analysis, four items changed from one indicator category to another, due to the change of one category from “Quality and reach of the teaching strategy” to “Organization of emergency remote teaching”.

In the elaboration of the last version, 17 items were merged with others because they were understood as sub-items, as they were related to or depended on the answer in another item.

After all the changes pointed out by the three focus groups and family representatives’ commentaries, as well as those research participants comments regarding the changes made by the researchers, the final ERT assessment instrument with the four categories of indicators and the six open questions is available at: https://drive.google.com/file/d/11wQoKSzneBVdU7mK9Fu-TYpsOwMhvK4R/view?usp=sharing (access on July 07, 2022).

Final considerations

The creation of instruments for the evaluation of ERT is an important movement in the construction of a body of reflections and academic knowledge that feed the understanding of potentialities, challenges and limits of remote teaching in times of crisis, when it is not possible to attend school. Due to other adverse events, it may be necessary to resort to emergency remote teaching in the future. In these situations, the improvement of ERT can profit from evaluations of previous uses of ERT, since, when Covid broke out in 2020, there were few ready resources, no pre-planned infrastructure, and little academic discussion, a context that characterized the emergency aspect of remote teaching after school closures (^{WHITTLE et al., 2020})

Among this research participants’ considerations, the suggestions regarding the language of the questionnaire stand out, since it could be difficult for some families to understand a question formulation using academic language, as was initially the case with some phrases and expressions.

Other changes in the questionnaire suggested by the participants referred to the order or placement of questions according to the indicator category to which they should belong and the exclusion of questions that were, in some way, related to the assessment of learning (since the research proposal is an assessment of the offer, context, processes and interactions, and not the results of the ERT).

On the other hand, we understand that in addition to the product (questionnaire; indicators), the very process of creating this instrument, especially in terms of the methodology used, constituted, for those involved in the focus groups (teachers, students, administrative workers, families), a significant moment of reflection on the ERT and on the different responsibilities of public policies, school and family in this mode of teaching, as well as on the possibility that some elements of the infrastructure and organization of ERT continue to be used when returning to face-to-face school activities, as could be the case of support for the teaching and learning, and school-families communication, an aspect highlighted by some of the research subjects.

This study and the created questionnaire have some limitations. Due to the peculiar characteristics of the school (infrastructure, teaching and technical staff, students and families) and the research methodology, the instrument cannot be generalized to state, municipal or private schools, although it is possible to imagine that several issues in the present instrument may be appropriate for ERT assessment in other schools. The questionnaire also does not allow the assessment of the learning effectively constructed by the students. The considerable number of questions is an aspect that can discourage the school community to complete the questionnaire. Another limitation is the absence of questions specifically designed for students with disabilities, although aspects related to this audience can be presented by the respondents, especially in the open questions.

Despite the challenges presented in the research process, we understand that the qualitative approach used was adequate to establish specific indicators for the investigated school, which may allow a qualified evaluation of its ERT. The questionnaire and the indicators, although they should not be reproduced for other school communities in other contexts, along with the academic reflection presented here, may serve as an inspiration for the creation of ERT evaluation in other schools and teaching networks.