Development of questions: rationale and reference matrix

When we refer to the construction of questionnaires, performance tests and other research instruments aiming to assess indicative of competencies, attitudes and skills through descriptors and variables, we must consider as a starting point the elaboration of a reference matrix to organize and describe in a specific way the assessed elements (^{PAGAN; TOLENTINO-NETO, 2015}; ^{ALVES, 2019}).

In this perspective, the research instrument, called the Digital Vulnerability Identification Questionnaire (DVI-Q) was elaborated together with a specific reference matrix, whose objectives, descriptors, and categories were previously established to guide the construction of the questions in a reasoned manner. The process of developing the first version of the DVI-Q, as well as its reference matrix, was based on content analysis of bibliographic research on related themes for the definition of constructs and dimensions that were addressed and organized into categories.

Initially, we used the contributions of Ayres et al. (2003) and Ayres, Paiva and França Júnior (2012) to define and understand the concept of vulnerability, which, in turn, is subjective and involves practical and epistemological foundations articulated with behavioral, cultural, political, and economic aspects to understand how certain population groups, in the analytical dimensions of individual, social and programmatic character, become exposed to health problems and risks. Indeed, the understanding of vulnerability presented in this research is inserted in a multidimensional sphere and related to biological, epidemiological, attitudinal, and socio-cultural factors.

To associate the theoretical contribution of vulnerability with the second central approach of the DVI-Q, that is, the inappropriate use of digital technologies, we resorted to the review of several studies focused on this theme, predominantly in the areas of health and education, and with different populations and methodological approaches (^{YOUNG, 1998}; ^{WIDYANTO; McMURRAN, 2004}; ^{KING; VALENÇA; NARDI, 2010}; ^{CONTI et al., 2012}; ^{KING; NARDI; SILVA, 2014}; ^{YILDIRIM; CORREIA, 2015}; ^{GONZÁLEZ-CABRERA et al., 2017}; ^{LEE et al., 2017}; ^{SILVA, 2017}; ^{LOUREIRO; GALHARDO, 2018}; ^{SENADOR, 2019}; ^{KING et al., 2020}; ^{SILVA et al., 2021}).

Among the studies mentioned, some have developed their own validated instruments to assess a range of phenomena related to technology abuse, such as the Internet Addiction Test (IAT), initially proposed by Young (1998) and improved by ^{Widyanto and McMurran (2004)} and which has since been translated and adapted to various contexts and even served as the basis for the creation of other instruments, such as the Nomophobia Questionnaire (NMP-Q), by ^{Yildirim and Correia (2015)}, which corresponds to another important tool for diagnosing disorders caused by pathological dependence on digital technologies. Besides these, with a look more focused on the contemporary national context, we highlight the contribution of King, ^{Nardi and Silva (2014)} and ^{King et al. (2020)}, along with other researchers and collaborators of the Delete Institute: digital detox and conscious use of technologies, of the Department of Psychiatry of the Federal University of Rio de Janeiro (UFRJ).

The questions of the DVI-Q were elaborated based on the mentioned studies, but with an educational perspective and without the intention of evaluating the cognitive learning about technological concepts, but, in a pluralist epistemological sphere, to identify possible relations of the use of technologies with different levels of vulnerability through a Likert-type scale with four points.

At first, the DVI-Q prototype had 40 questions organized equally into four, specific, but at the same time interrelated categories, namely: Data exposure; Technology dependence; Physical/emotional aggravation; and Virtual alienation. A breakdown of these categories is presented in Chart 1, below.

In addition to the reference matrix², the DVI-Q questions were prepared using some criteria so that a group of specialists could establish their judgment during the process of content validation also through a Likert-type scale of four levels of agreement, in which the score 1 (one) refers to the total disagreement, and 4 (four) shows the total agreement of the item evaluated against the established criteria: characterization; relevance; vulnerability; neutrality; and objectivity. The details of these criteria are presented in Chart 2, below.

The establishment of these criteria contributed to systematize the evaluation of the experts, and, in addition to serving as a barema (set of tables or numerical data that present the result of certain calculations.), it also corresponded to the prediction potential of the questions, thus ensuring higher quality in the formulation of the items in order to avoid some biases, such as excessive use of technical terms and inappropriate language that could confuse or induce the respondents' answers.

Methodology, ethical aspects, expert panel formation, and pilot application

The validation of the DVI-Q followed the ethical principles established by the National Health Council (NHC), according to Resolution No. 510/2016, which deals with research with human beings, upon consideration and approval by the Research Ethics Committee (REC) of the Federal University of Sergipe, under the Certificate of Ethics Appreciation Submission (CEAS) No. 35407520.4.0000.5546 and Opinion No. 4,490.395. The social actors participating in this process are basically classified into two groups, the first composed of a panel of experts involving teachers/researchers and health professionals, and the second, a sample of students, and both contributed specifically to the instrument validation processes.

Initially, for the content validation process of the instrument developed, the Delphi method was used with the purpose of obtaining a grounded consensus among a group of specialists in relation to the contents, categories, descriptors, and criteria submitted to the evaluation. The choice of this method proved relevant because it is one of the main psychometric techniques used to form a panel of experts and, subsequently, to determine consensus for defining competencies, educational content, planning pedagogical actions, and establishing criteria and evaluation methods in questionnaire validation processes (^{PIRES; BRANDÃO; SILVA, 2006}; ^{JESUS, 2013}; ^{ANTUNES, 2014}; ^{MARQUES; FREITAS, 2018}; ^{ALVES, 2019}).

The participants who formed the panel of experts working in the processes of content validation of the DVI-Q were invited through the use of the snowball technique, being predominantly teachers/researchers who work and develop research involving the use of technologies in education and with experience in the development of questionnaires and school performance tests, as well as health professionals, also researchers, who work or identified with the themes addressed in the instrument, given the important relationship of the questions with the physical, social, emotional and other biopsychosocial and socioecological dimensions of health.

In fact, the snowball method corresponds to a non-probabilistic sampling technique that uses chains of references to define and reach the number of participants according to their accessibility (BIERNACKI; WALDORF, ¹⁹⁸¹). In this perspective, Rowe, and Wright (¹⁹⁹⁹), ^{Powell (2003)}, ^{Jesus (2013)}, and Alves (²⁰¹⁹) reinforce the importance of seeking a heterogeneous background among the participants who will compose the panel of experts acting in validation processes through the Delphi method to ensure balance and greater impartiality of judgments about the criteria evaluated.

It is worth remembering that, given the pandemic scenario in which the study was conducted, and in compliance with the health measures established by the competent bodies to reduce Covid-19 contamination, the entire validation process occurred remotely through MDICT. Thus, soon after the first version of the DVI-Q was prepared, the questions were transposed and adapted to the Google Forms platform, also known as Google Forms, which, being an online and free tool for creating and storing forms in the cloud; its use was notably intensified in scientific research and remote school activities during the pandemic period.

The evaluators who made up the panel of experts, which we can also call judges, as well as the group of students, were invited to participate in the research by free and spontaneous will, demonstrated through an Informed Consent Form (ICF), written in the form of an invitation letter and sent by e-mail, Telegram and WhatsApp, along with the link to access a questionnaire to know the sociodemographic profile of the participants.

It is noteworthy that the use of the Delphi method within the context of physical and social distance proved advantageous as it enabled the questionnaire and the other instruments to be evaluated by the specialists in more than one round, without them being gathered in the same environment, until the intended consensus was obtained. Thus, the process of repeating rounds had the purpose of reducing divergences and reaching a consensus of at least 80% among the evaluators about the items that compose the DVI-Q, as recommended by the literature (^{WESTMORELAND et al., 2000}; ^{JESUS, 2013}; ^{ANTUNES, 2014}; ^{MARQUES; FREITAS, 2018}).

After the conclusion of the Delphi method with the satisfactory consensus obtained by the evaluating judges, the DVI-Q was also submitted to a semantic analysis with the target audience for which this instrument is intended, i.e., a representative sample of students finishing the 3rd year of high school, so that they could explain their impressions about the questionnaire and point out the need for changes to better understand the questions and their applicability.

The use of MDICT proved to be an important and effective strategy in these stages, especially to mediate the communication between researchers and participants, ensuring access to the instruments and data collection without restrictions of time and space. On this discussion of the use of the Internet and other technologies in research, Flick (2009, p. 32) corroborates by arguing that "many of the existing qualitative [and quantitative] methods are being transferred and adapted to research that uses the Internet as a tool, as a source, or as a research question.

In this sense, after completing all the steps of the content validation process of the DVI-Q, in a second moment, this time in person, we submitted this instrument to a pilot application to a larger sample of students, in order to statistically measure the reliability of this questionnaire by analyzing its internal consistency in the Statistical Package for Social Science (SPSS) software, as well as to allow correlations between the variables and with other research instruments.

Results and analysis of the instrument validation process

The Delphi method allowed the group of specialists to assess the DVI-Q questions both quantitatively, using the Content Validity Index (CVI) about the judges' level of agreement, and qualitatively, based on the considerations, orientations, and suggestions sent by the evaluators on the forms. Although Coluci, ^{Alexandre and Milani (2015}, p. 931) have not found in their review study a consensus in the literature that characterizes a particular statistical test as the gold standard for content analysis, it was evidenced that, in addition to the qualitative approach through the considerations of the panel of experts, it is important to perform a quantitative analysis through the CVI to measure the "proportion or percentage of judges who agree on certain aspects of the instrument and its items".

In this context, the CVI was calculated using the Likert scale, in which the descriptors were evaluated in four levels, where level 1 (one) was considered not relevant or not representative, since the specialists indicated total disagreement with the evaluated item, while the maximum level, that is, 4 (four), refers to the relevance and representativeness, in view of the total agreement of the evaluators about the item. 80% favorável na análise entre os juízes (ou seja, IVC> 0,80), conforme recomendação dos estudos de ^{Polit e Beck (2006)}, ^{Alexandre e Coluci (2011)}, ^{Revorêdo et al. (2016)}

For the descriptors and criteria evaluated in the reference matrix and in the DVI-Q to be considered valid or satisfactory, they had to obtain a minimum percentage of agreement of 80% favorable in the analysis between the judges (i.e., CVI ≥ 0.80), as recommended by the studies of Polit and Beck (2006), ^{Alexandre and Coluci (2011)}, ^{Revorêdo et al. (2016)}, and ^{Souza, Alexandre, and Guirardello (2017)}. Thus, considering that the CVI corresponds to the ratio between the number of items considered satisfactory or adequate by the experts and the overall quantity of items, we can represent this equation to evaluate each item individually through the following formula:

It was necessary to tabulate quantitative data to measure the percentage of agreement of the judges' assessment, i.e., the CVI, about the questions before the established criteria (as shown in Chart 2), whose answers marked as "Strongly disagree - (1)" and "Disagree - (2)" were considered unsatisfactory, as well as those marked as "Agree - (3)" and "Strongly agree - (4)", which were considered satisfactory on the Likert scale. In summary, each question to be considered valid had to obtain the minimum 80% agreement percentage (CVI ≥ 0.80) in all five criteria established.

The questions that obtained a CVI below 0.80 were reformulated based on the experts' considerations and submitted to a second round of evaluation. However, some recommendations about the remaining questions with favorable CVI were also considered for analysis and contributed to further improve the quality of these items. In this sense, besides the CVI of each criterion evaluated per question, we also calculated the overall CVI* of each category. Table 1 below presents the results for the category Data Exposure.

Although the overall CVI of the Data Exposure dimension was considered satisfactory, with a value of 0.84 according to the evaluation of the panel of experts, it is noted that eight questions were evaluated as unsatisfactory because they did not meet any of the criteria assigned for the judgment, being the item "Neutrality" the most pointed out in need of reformulation, specifically in questions DE1, DE2, DE3, DE4, DE5, DE7, DE8 and DE9; In addition to this criterion, "Relevance" in question DE7 and "Objectivity" in question DE8 were also considered unsatisfactory.

In this perspective, we anticipate that the most commented criterion among the four categories and that, to some extent, also divided opinions before the evaluating judges was that of "Neutrality". However, we emphasize that the purpose of this question would be to identify whether the wording of the question would present any biased element capable of influencing the direction of the respondent's answer, since the wording of the alternatives was purposely prepared with value judgments to identify what would be the possible opinions and attitudes that the respondents could demonstrate by marking their answers.

Given the above, we emphasize the important contribution of judges 18PSI_MEST and 25LET_MEST on this question, reinforcing how the vision of the evaluators may be different compared to adolescents, the target audience of the DVI-Q.

Judge 18PSI_MEST:I consider the sentences to be well prepared, with accessible language and consistent with the audience to be researched. Besides, they are compatible with the proposed objectives and investigated indicators.

Judge 25LET_MEST: I liked the questions, but in relation to neutrality I put I always agree, I will justify because, as I work with rhetoric and argumentation, it is [...], I do not consider that there is anything neutral or impartial, there is always a bias, whether ideological or something like that that tries to convince or persuade a certain audience about what is desired [...]. For me, reading the questions, they are all presented very ironically, but of course this questionnaire is not meant for me, but it will be for teenage students, and they may have another view. And I believe that they will have another view and will not realize it, because we realize, for example, like this... I don't see the need to leave my profile private on social networks, so, at my age and in view of everything we read and the things we know, we end up finding it ironic, but for them it wouldn't be. That's why I put this neutrality [...], it is neutral from their point of view, but from our point of view, that we are evaluating the issues, I don't consider it neutral, so I'm just justifying it.

From this perspective, it was necessary to make several changes to the wording of the questions based on the judges' considerations, and we will bring some of these narratives to exemplify, or even justify, the changes made.

Judge 07QUI_MEST: I consider that questions DE1; DE2; DE3; DE4; DE5 and DE8 do not present Neutrality in their wording because by using the term "No" it denotes something at least doubtful and that can influence the student's answer. It generates fear, insecurity and influences the answer.

Judge 25LET_MEST: In question DE2, I would complete, although the question was not bad, I would add for the sentence to be complete [...] "for the more followers, the better", so the question will not have ambiguity, the more invitations, or the more followers, right? Although it is simple and I am understanding that it refers to followers, however, since it is for high school student, I think it is better to make it clearer.

For consultation purposes, we present in Figure 2, below, a QR Code that will give access to the comparative charts with the wording of the questions both in the first version and after their reformulation during the validation stages.

In Table 2, below, we present the percentage of agreement of the panel of experts, calculated by means of the CVI of each criterion per question, referring to the Technological Dependence category, as well as the overall CVI*.

Although the Technological Dependence dimension was considered valid against the overall CVI of the experts' judgment, seven questions were evaluated with the "Neutrality" unsatisfactory. Although we have already discussed the rationale behind this criterion, the judges' recommendations were taken into consideration to reformulate some of these questions, as we can see below.

Judge 08LET_D_IN: I noticed that in some questions the "neutrality" question was compromised, in the sense that some value judgment was "embedded" in it. The other questions, "characterization, vulnerability, relevance and objectivity" are all very well identified.

Judge 25LET_MEST: This second category, I thought it was very important, and even we end up recognizing ourselves in these parts [laughs], because we do have a certain dependence on the use of technologies, I [...] honestly I realized that there are a couple of questions here that went directly to me, because I have my cell phone all the time, sometimes I'm not using it, I leave the internet off, but the cell phone is right there by my side, even with the internet off, to study and so on.

The narrative of the judge 25LET_MEST points to strong indications that this instrument can also be suitable for adult audiences, such as academics, teachers, and the general population, as long as cross-cultural adaptations are made to align the questions to the characteristics of certain population groups.

However, considering some criticism, the judge 19BIO_D_IN pointed out that by answering some questions, the adolescent would not necessarily be indicating pathological dependence on technologies, and we agree with this consideration, and this is not even the goal of the questions, since pathological dependence on technologies concerns a clinical diagnosis, and even has a specific terminology, nomophobia. It is worth noting that, in the face of the current globalized scenario, we consider that all people have some dependency relationship with technologies, whether for entertainment, work, study, etc., so the focus of this dimension is to investigate whether the respondent has indicators of overuse of these devices for various factors, but without entering this specific context of the health area.

In Table 3, we present the percentage of agreement of the panel of experts through the CVI concerning the criteria evaluated in each question of the category Physical/Emotional Aggravation, as well as the overall CVI* of this category.

It is again observed that the category "Neutrality" remained the most resonant among the judges compared to the others, in which questions PE6, PE7 and PE10 were evaluated as unsatisfactory, but, overall, the category Physical/Emotional Aggravation was considered valid, visualizing the CVI* of 0.87.

Judge 25LET_MEST: Regarding these last two categories, I find them very interesting because it has to do with this perfection that is propagated on social networks and ends up causing a certain feeling of sadness, depressive feelings to the point that the person ends up thinking that their life is not, in quotes, the life that people on social networks have, right? So, I think it is important to address this in the questions, because if the young person or any other person evaluates their life according to what they see on the social networks, it is also a question of consumerism, if they are influenced, I don't know if this is the objective of the research, but I think it is useful for everything, I have even talked to some friends about this, that sometimes we see couples, we see people always smiling, always traveling, and we end up thinking, oh, my life is so boring, my life has nothing so good about it. So sometimes we end up, unconsciously or consciously, comparing ourselves to these perfect lives that are shown in the social networks, but which in fact is not that, and I believe that the same can happen with adolescents.

Finally, in Table 4, we present the quantitative data of the judges' judgment about the questions belonging to the Virtual Alienation category, in which, again, the percentage of agreement of these experts before the criteria of each item is represented by the CVI, as well as the general CVI* of the category.

Although the overall CVI of the Virtual Alienation category was also considered satisfactory, again the criterion "Neutrality" was evaluated as unsatisfactory in three questions, VA2, VA7 and VA8, and after the reformulation according to the judges' considerations, the questions were submitted to a second round of evaluation. This second round occurred exclusively through WhatsApp and Telegram within seven days, in which the evaluators could compare the questions before and after the changes based on the opinions raised by the group.

With the analysis of the new considerations of the judges, we found the need to significantly change only three questions, all belonging to the dimension Data Exposure, and after all the reformulated questions of the DVI-Q were considered valid, the next step corresponded to a pilot testing of this instrument, in a commented way, with a sample of the target audience to further qualify the semantic evaluation of the questions.

This new stage of semantic validation had the voluntary participation of 11 (eleven) students concluding the 3rd year of high school, enrolled in a school belonging to the Sergipe state education network, and occurred in remote mode, in which the questions were previously made available through Google Forms to be analyzed and, subsequently, a collective meeting was scheduled synchronously through the Google Meet platform, lasting approximately 40 minutes.

All students participating in this stage were over 18 years old and, therefore, could confirm their participation through the informed consent sent along with the instrument. As for the gender and age distribution, seven students were female, aged between 18 and 20. and the remaining four were male, aged between 18 and 22.

In general, this stage basically aimed to verify if the DVI-Q was really understandable to this sample of students, that is, if the items offered clarity and objectivity, as well as to identify any problems of ambiguity and other inconsistencies that could compromise the understanding of the adolescent audience. During this process, the students were asked about the rationale behind the questions, whether they were clear, or whether there were any terms they did not understand, as well as about the presence of any items or passages that did not make sense or that bothered them, and whether they had any comments or suggestions to make.

At the end of this process, it was found that the students showed good acceptance to the DVI-Q, with no reports of comprehension difficulties and no other factors that could cause any embarrassment or discomfort to them. However, most of this audience showed some discouragement with the number of questions, that is, 40 (forty) questions.

It is worth mentioning that this concern had already been alerted in the initial stage of evaluation by some judges, but we chose to keep the quantity for this moment with the purpose of knowing the opinion of these participants about which would be the most relevant items and that should be kept in the final version of the instrument. Thus, the students were asked to indicate four questions from each category to be excluded from the DVI-Q, which, at first, resulted in differences of opinion when asked why the questions they indicated should be excluded. To shed some light on this step, here are some fragments of the discussions.

Student 07: I saw no problem with the questions, no, on the contrary, I thought it was cool to answer them, like this, there are a couple of questions that I have already committed, like spreading news without knowing if it is true, installing viruses on the cell phone and having to format it, and even this thing of intimate photos, even without showing the face it is still a danger if the cell phone falls into the wrong hands [...]. Another thing, it's not my case, but I know a lot of people who get sick when they can't get good reviews on the internet, I don't care about that, but I have friends who practically beg us to follow them, like their pictures and comment on them.

Student 09: [...] it is very difficult to choose which questions should be excluded, I thought about this one about nudes, because I have never sent nudes and I don't even think of sending them, but after you asked me if I have any friends or know someone who has already shared this type of photo, I remembered some of my friends who have already done this and had a bad experience, so I changed my mind, I think it is an important subject to think about [...]. This question five of the third category [PE5], in my opinion, I would remove from the questionnaire because it is very similar to the first one of the previous category [TD1], and this last one that talks about addiction, shortness of breath and fast heartbeat [TD10], I also think it is important, but the question was too big, if I could make it smaller, it would be better.

It was possible to notice, based on the students' narratives, that the DVI-Q was well accepted by the target audience. The biggest difficulty was to reach a consensus about which questions should be excluded, but after some questions about why certain questions should remain and others not, new reflections and debates were raised, and, finally, most students agreed to eliminate the following questions: DE1, DE5, DE7 and DE10 (Data Exposure category); TD2, TD3, TD4 and TD6 (Technology Dependence category); PE3, PE5, PE9 and PE10 (Physical/Emotional Aggravation category); and VA2, VA4, VA6 and VA8 (Virtual Alienation category). This step also allowed making new changes in some questions, making them clearer and more consistent with the reality of this public.

In this context, ^{Hill, and Hill (2012)}, ^{Cunha (2015)} and ^{Alves (2019)} corroborate when they point out that this semantic validation stage contributes to qualify the instrument with even more relevant questions, allowing to assess the degree of understanding, clarity, and comprehension of the statements. Thus, the final version of the validated DVI-Q had 24 questions evenly distributed into four categories, as shown in Table 4.

Pilot application, reliability, and interpretation of the DVI-Q

After completion of the processes of content validation of the DVI-Q, the next step corresponded to the pilot application of this instrument, along with a socioeconomic questionnaire, which were answered by 3rd year high school students enrolled in four schools of the State Secretariat of Education, Sports, and Culture of Sergipe (SSESCS) in October and November 2021. To ensure statistical reliability, the choice of regional directorates and the respective school units invited to carry out this stage of the research was made by means of probability sampling techniques of the simple random type, in the perspective of ^{Barbetta (2007)}, through an unbiased draw.

It is important to emphasize that, although the Statute of the Child and Adolescent (SCA) considers adolescence a complex process of growth and biopsychosocial development within the age range of 12 to 18 years, and may extend under some conditions to 21 years, as well as the Statute of the Youth classifies as young people those aged between 15 and 29 years, we prefer to adhere in this study to the WHO classification that circumscribes adolescence to the second decade of life, i.e., from 10 to 19 years of age, because, in addition to chronological criteria, we consider that biological, psychological and social factors are also important and should be valued in this conceptual approach (^{BRASIL, 1990}. ²⁰⁰⁷, ²⁰¹³).

In this context, the grand total of students with valid answers included in this research corresponded to 147 participants. This number is considered satisfactory to perform statistical processing and analysis in SPSS. A balance was noticed regarding the gender distribution of the participants, since the sample consisted of 53% (n 78) of female students and 47% (n 69) of male students, and, regarding the age range, the minimum age corresponded to 16 and the maximum to 19 years, with a mean of 17.37 years and standard deviation of 0.907, prevailing the range between 17 and 18 years, with a frequency of 45.6% (n 67) and 25.2% (n 37), respectively

Regarding the analysis of the internal consistency of the surveyed instruments, it is worth mentioning that reliability corresponds to a statistical measure of the reproducibility or stability of the data collected in a consistent manner in time and space. Thus, based on the assumption that any assessment instrument involving data collection is subject to errors of inaccuracies and other biases, it becomes necessary to test its accuracy through reliability techniques (^{PASQUALI, 2009}; ^{JESUS, 2013}; ^{ALVES, 2019}).

Considering that the reliability analysis is a necessary condition for the research instruments to present consistent results about what they propose to measure, and with the least amount of bias possible, among the numerous techniques used, Cronbach's alpha coefficient is one of the most used due to its vast consolidation and contribution in the academic and statistical circles. Thus, all responses collected from 147 participants were imported into a database in SPSS software and submitted to statistical tests to evaluate the homogeneity of the DVI-Q through Cronbach's alpha coefficient, thus allowing an analysis of the internal consistency of the questions. This coefficient allows analyzing the profile of the answers provided by respondents to calculate the reliability between the items of the questionnaire, whose values range between 0 and 1, in which, the closer to 1, the greater the reliability of the dimensions of the construct (^{CRONBACH, 1951}; ^{PASQUALI, 2009}; ^{SILVA, 2017}).

It is important to note that the level of acceptability of this coefficient depends on the type of research conducted, but in general, Landis and Koch (1977) classify indices between 0 to 0.20 as small or unsatisfactory consistency; 0.21 to 0.40 as reasonable; 0.41 to 0.60 moderate; 0.61 to 0.80 substantial; and 0.81 to 1.0 almost perfect. However, the most recent literature points out that values above 0.80 are considered ideal for research in the health sphere, while for studies related to the social sphere Cronbach's alpha values above 0.60 are reliable (^{STREINER, 2003}; ^{HILL; HILL, 2012}; ^{ALVES, 2019}).

In this context, in order to analyze the internal consistency of the DVI-Q, we initially calculated Cronbach's alpha coefficient in general with all 24 questions, and obtained the index α = 0.821 (based on standardized items), considered satisfactory, which shows the existence of internal consistency of this research instrument, thus being possible to verify its validity in the qualitative-quantitative aspect and create the latent variable "Indicators of Digital Vulnerability" for the population studied. When we calculate this coefficient separately with respect to the categories presented in ^{Chart 1}: "Data Exposure", "Technological Dependence", "Physical/Emotional Aggravation", and "Virtual Alienation", in isolation, although all categories have presented satisfactory indices (α > 0.600), we realize that they have better consistency and reliability when applied together, as we can see in Table 5, below:

Considering that, although the DVI-Q has four blocks of categories subdivided into six questions each, this instrument was designed to have its full applicability with all 24 items, and for this reason, the results obtained through Cronbach's alpha coefficient are considered satisfactory and consistent with what was expected, thus meeting the purpose of its development. Although the target audience of this version of the DVI-Q corresponds to adolescent high school students, it was possible to infer, through psychometric analyses, that this instrument, through cross-cultural adaptation processes, may also have its applicability validated for different audiences, and may even be applied along with other instruments to investigate various correlations through statistical analyses, according to the purpose of the applicators, thus contributing to new research related to the use of digital technologies.

Chart 4shows the final version of the DVI-Q after all validation stages were performed: content validation, by means of the judges' evaluation using the Delphi method; qualitative validation, by means of semantic analysis with the target audience sample; and quantitative validation, resulting from the pilot application and statistical analyses to calculate internal consistency and reliability.

Regarding the different analytical and interpretative possibilities of the DVI-Q in light of the literature that guided its preparation, we defined that, once answered, the results may be established in two ways according to the score obtained, "general" and "by categories", with the following levels of indicative of digital vulnerability: absence of vulnerability; mild vulnerability; moderate vulnerability; and high vulnerability. We present in Chart 5, below, the classification of these levels according to the respondent's score.

For disclosure purposes, we present in Figure 3, below, a QR Code that will give access to the prototype of the digital version of the DVI-Q compatible with different interfaces and digital devices; however, we point out that this version will be improved soon, with the linking of a database so that it can be used in new research.

The analysis of the results obtained with the application of the DVI-Q, both in general and fragmented by categories, will make it possible to establish a wider interpretative range of these results in a qualified and systematized way, because it is possible, for example, that a certain student does not show an alarming level of digital vulnerability in the general score of the questionnaire, but, when investigating the results of each category, it is possible to see that this same student showed himself vulnerable in some of them, thus allowing the teacher to intervene educationally with strategies aimed at the conscious and responsible use of technologies.