2.1 Evaluation of higher Education in Brazil

In Brazil, the rapid expansion of higher education at the end of the 20th century led to the need to implement evaluation systems to ensure quality levels of professional and academic training (^{PEIXOTO, 2009}). With the growth in the number of HEIs, especially the private ones and, consequently, the reduction of the State's presence in educational funding, the exams gained notoriety and importance as quality control instruments (^{DIAS SOBRINHO, 2010}). Thus, the evaluation of Brazilian higher education was more effective from the 1980s, going through different experiences and gradual evolution.

^{Polidori (2009)} described the evolution of the development of higher education in Brazil in four cycles comprised in the periods: (i) from 1986 to 1992, in which several initiatives to organize an evaluation process and the existence of isolated evaluations in the country did not constitute a national assessment; (ii) from 1993 to 1995, called policy formulation and Installation of the Institutional Evaluation Program of Brazilian Universities; (iii) from 1996 to 2003, called the consolidation or implementation of the government proposal, and in which the development of the National Course Examination, the Provão, and the Evaluation of Offer Conditions, later called the Evaluation of Teaching Conditions; (iv) from 2003 to the present, called the construction of emancipatory assessment, with the implementation of Sinaes, in a proposal to develop a formative assessment that considers the specificities of HEIs in the country.

In force since 2004, by Law n. 10.861/2004, Sinaes is the current higher education evaluation system created to create an integration between evaluation and regulation (^{PEIXOTO, 2009}). He proposed a global assessment integrating all aspects of HEIs and various application instruments to comply with their components (^{PEIXOTO, 2009}; ^{DIAS SOBRINHO, 2010}).

Regarding evaluating students' performance, Enade was instituted with the purpose of a dynamic evaluation that proposes to consider the change and development of the student during the course (^{DIAS SOBRINHO, 2010}). Enade is a mandatory component and is included in the students' diplomas. Data collection takes place through the following instruments: (i) test, intended to assess the student's performance concerning the syllabus provided in the course's curricular guidelines, their abilities to adjust to the demands arising from the evolution of knowledge and their skills to understand issues outside the scope of their profession; (ii) student questionnaire, designed to gather information to characterize the profile of students and the context of training processes; (iii) test perception questionnaire, intended to gather information that allows assessing the students' perception about the instrument; and (iv) the course coordinator's questionnaire, designed to gather information to characterize the profile of the course coordinator and the context of the training processes.

It is noticed that evaluation in an institution is one of the main tools for implementing and organizing educational policies and reforms. It causes changes in teaching practices, curricula, management model and structure, research priorities and the entity's social responsibility (^{DIAS SOBRINHO, 2010}).

2.2 HEI Management and the Role of the undergraduate course coordinator

Over the last 40 years, HEI management has become based on rational models inspired by the reality of companies. However, such models did not generate compelling results, as they were institutions in which the management process involved rational, political and symbolic aspects (^{MEYER JÚNIOR, 2005}). In this context, it was understood that the management of HEIs must be differentiated from other types of organizations for several reasons, including the mission. While the assignment of a business organization is easily identified, as they are aimed at profit, HEIs have the mission of providing teaching, research and extension (^{BARBOSA; MENDONÇA, 2016}). Thus, the mission of an HEI is more complex and subjective, requiring more management work.

^{Rebelo and Erdmann (2007)} point out that among HEIs, universities are more complex organizations with multiple relationships, which must be evaluated in all their specificities. Understanding corroborated by ^{Silva and Cunha (2012)}, HEIs (public, private or community) are organizations composed of particularities and belong to an economic segment differentiated from business organizations. Furthermore, business techniques tend to value the quantification of results, which would prioritize the number of enrolled or graduated students (for example), but without assessing the effectiveness of the teaching-learning process (^{SILVA; CUNHA, 2012}).

In this context, ^{Meyer Júnior, Pascucci and Mangolin (2012)} point out that HEIs are no longer inserted in a peaceful environment, where students arrive continuously and increasingly, functioning in a climate of stability. More recently, given the increase in HEIs, private institutions, for example, have inserted strategic planning to improve management as a way to guarantee short-term viability and long-term permanence.

The elaboration of a strategic plan, with a political and institutional character, serves as a reference and justification for the main actions of the institution, in addition to legitimizing the management's intentions before the academic community. However, in the conception of ^{Meyer Júnior, Pascucci and Mangolin (2012)}, the practice of strategic management in HEIs has revealed a gap between formal plans and the effective practice of strategies due to the differences in contexts between business and educational organizations and, because programs and models created for companies are unlikely to be useful for HEIs. In compliance with this, ^{Ferronato (2017)} highlights that implementing Sinaes and its adaptations implies a management model for HEIs, since the system indicates actions and strategies that make up the indicators for the final evaluation. Thus, considering that an HEI is premised on the quality of education offered, managers, especially course coordinators, now significantly contribute to the institution's management. The coordinators act as catalysts for numerous processes and activities, as they immediately control the courses (^{BOTELHO et al., 2018}).

The undergraduate course coordinator's role, duties and profile may vary according to the HEI. ^{Bonzanini, Ferreira-da-Silva and Leite (2017)} explain, for example, that in public HEIs, in addition to administrative attributions, there is a character more focused on research and academic activities, while, in private and community HEIs, functions are more characteristic of management, since these purpose at results that should contribute to their sustainability. However, regardless of the nature of the HEI, the coordinator's main attributions are to comply with the legislation, accompanied by institutional commitment, in line with the Course Pedagogical Project (CPP) provisions.

Thus, the coordinator is perceived as the articulator between professors and students in the teaching-learning process, seeking to make the course objectives viable through pedagogical mediation (^{DOMINGUES et al., 2011}). He is the conductor of qualifying the education offered, as he must monitor the development of students, aiming to train qualified professionals. However, like the other HEI managers, the course coordinator is a professor who, according to institutional norms, can take over by appointment or election (^{COSTA, 2009}). When assuming the coordination, he will have a new identity, that of a manager, but without ceasing to be a teacher (^{MARCON, 2011}). ^{Botelho et al. (2018)} point out that many assume they need to gain experience in a similar position since they are professors in different areas. Those who take the role without knowledge present, at first, the behavior of professors concerned only with the classroom; those with previous experience in managerial positions, however, current behaviors characteristic of managers, in addition to those of teaching (^{MARCON, 2011}).

Given the competitive scenario in which HEIs are inserted, improvisations or doubts regarding the activities to be developed are not allowed. Thus, more and more qualified and prepared professionals are required to occupy management positions at the university level (^{BARBOSA; MENDONÇA, 2016}). Therefore, HEIs and course coordinators need to use indicators capable of expressing their limitations and potential to improve the management process.

Given this, this study proposed to build a set of indicators that allow each undergraduate course to assess whether its strategic actions contribute to improving students' performance in Enade. With this focus, the indicators were proposed observing as the main object of the Assessment of Undergraduate Courses instrument, which focuses on identifying the teaching conditions offered to students, in the dimensions of the faculty, the didactic-pedagogical organization and the infrastructure (^{BRASIL, 2004}). In addition, the methodology and evaluation indicators were also observed in the postponed, according to the contributions of ^{Zanin (2014)} and ^{Molozzi (2015)}. For this, the model was divided into five axes: didactic-pedagogical indicators, adopted methodologies, evaluation, faculty and infrastructure.

3.1 Population and sample

The population of this research comprised 547 coordinators of the most different undergraduate courses of 14 HEIs, linked to the ACAFE System, maintained by foundations instituted by municipal law in the State of Santa Catarina. ACAFE, founded in 1974, aims to promote the integration of HEI consolidation efforts, carry out technical-operational support activities, and represent them before state and federal government bodies. It currently integrates 16 HEIs, 11 universities and five university centers (^{ACAFE, 2020}).

For sample selection, two institutions were excluded from the population. The first is the State University of Santa Catarina, a foundation created by the State of SC, thus characterizing itself as a state public university. The second is the Municipal University Center of São José, maintained by the Municipal Educational Foundation of the city of São José, characterizing itself as a municipal public institution. The exclusions are justified by both having structural organization and management models differentiated from the others due to their public character, and by the fact that the budget comes from the Public Sector, while the others, even if instituted by municipal law, are not necessarily linked to the respective municipal governments, having their management, with independent councils.

Among the list of institutions participating in the research are: the Regional University of Blumenau; University of Contestado; University Center – Catholic of Santa Catarina; University of the Extreme South of Santa Catarina; Barriga Verde University Center; University Center for the Development of Alto Vale do Itajaí; University Center of Brusque; University of Planalto Catarinense; University of Southern Santa Catarina; the University of Vale do Itajaí; University of Joinville Region; Community University of the Chapecó Region; University of West Santa Catarina and the University of Alto Vale do Rio do Peixe.

The final research sample comprised 110 undergraduate coordinators from the 14 participating HEIs, equivalent to 20.11% of the population. The questionnaire was sent to the coordinators through an electronic link (Google docs.), via email from December 2017 to February 2018.

3.2 Research tool

The research instrument was elaborated in two parts. The first consisted of 14 questions related to the respondents' profile, and the second initially proposed 31 indicators, subdivided into five sets, related to the improvement of Enade’s indicators.

To validate the questions, the questionnaire was sent for analysis by five specialists in the area, four professors from different HEIs, and one professional linked to Ministry of Education with a management position related to Enade. Of these, only one evaluator still needs to return his suggestions. However, it should be noted that one of the evaluators who responded invited another professor who is also a reference in the area to contribute to the analysis, and with that, there were a total of five respondents.

After this first stage, a pre-test was carried out with three coordinators with experience managing undergraduate courses which were not participants in the research. This step aimed to assess whether the proposed model and how the questions were presented were adequate and easy to understand. The reviewers considered that it was following the proposed objectives and could be forwarded to the respondents.

After the steps above, the instrument was finalized with 51 questions, 14 of which were about the profile of the respondents and 37 related to the improvement of the Enade indicators. The profile questions dealt with age, gender, area and time of training, area of expertise, previous professional experience, time at the institution and acting in the course coordination, weekly workload and participation in management-related training. As for the questions about the improvement of the Enade indicators (Table 1), they were presented to verify the degree of importance assigned (Likert scale, 1 being unimportant and 5 being very important) and the degree of execution (Likert scale, 1 being not executed and 5 being executed very often).

3.3 Análise dos dados

For data analysis, descriptive statistics were used, and information entropy calculation. Entropy is considered a simple but important measure, due to the amount of information it provides (ZELENY, 1982). ^{Rocha (2010)} describes the calculation of informational entropy as shown below.

Let di=(di1,di2,…,dim) the values be normalized, where: dik=xikxi , characterizes the set D, in terms of the ith attribute. Lies Di=∑k=1mdik;i=1,2,…,n .

The entropy measure of the intensity contrast is sought for the ith attribute calculated by e(di)=−α∑k=1mdikDiLn(dikDi) , where α=1emax>0 e emax=Ln(m) . It is also observed that 0≤dik≤1 e dik≥0 . If all dik are equal for a given i, then dikDi=1ne e(di) assumes the maximum value, that is, emax=Ln(m) . When fixing α=1emax , it is determined 0≤e(di)≤1 for all e(di)s where this normalization is necessary for comparative purposes. In this way, the total entropy of D is defined by: E=∑i=1ne(di) . The larger e(d_i), the smaller the information conveyed by the ith attribute. If e(di)=emax=Ln(m) , then the ith attribute does not convey information and can be removed from the decision analysis. Because weight λ~i is inversely related to e(d_i), use 1-e(d_i) instead of e(d_i) and normalize to ensure that 0≤λ~i≤1 e ∑i=1nλ~i=1 Thus, the information entropy can be represented by:

Next, the analyzes and discussion of the results are highlighted.

4.1 Results

Table 2 presents the profile of respondents participating in the survey.

Regarding age, it was observed that more than 60% of coordinators are between 36 and 55 years old, and that women are predominant at the head of coordination, representing 55.45% of the sample. Concerning professional experience before the position, it was found that 45 of the respondents stated that they had not had previous experience in any area of management, while another 45 claimed to have expertise in the area of administrative management, and another 20 claimed to have experience in the area of academic leadership. In addition, the length of time working in the management area was verified, in which it was found that the majority (39.36%) stated that they had been working in the management area for between 5 and 10 years.

About the time of work in the institution, it was evidenced that the highest concentration is between 5 and 10 years (n=34), followed by those with less than 5 years (n=23). When asked about the time spent coordinating the course, the highest concentration (30.91%) has between 2 and 4 years, followed by those with less than 2 years (23.64%). Regarding the workload dedicated to the institution, it was found that most have a workload between 30 and 40 hours per week. Regarding the workload dedicated to coordination, the majority (69.09%) dedicate more than 16 hours per week to the function.

As for training, it was found that the respondents were instructed in 41 different courses, which, to facilitate the analysis, were organized according to the Areas of Knowledge established by CNPQ. The highest concentration of respondents was observed in the Social and Applied Areas (32.74%). This area includes courses in Administration (12), Architecture and Urbanism (5), Accounting (7), Social Communication (1), Industrial Design (2), Law (3), Economics (3), Journalism (2), Publicity and Propaganda (1) and Social Service (1). The second most representative area was Human Sciences, with 21.24%, with courses in Geography (2), History (2),

Pedagogy (11), Psychology (8) and Sociology (1). It is also noted that the majority followed in the same area to take their postgraduate courses (master's and/or PhD).

Finally, regarding participation in training courses related to management, the majority (95.45%) mentioned participating. Regarding the frequency with which they participate in these training courses, 47 respondents said that they experience at least once a year; 49 claimed to participate in 2 to 5 trainings per year, 9 respondents indicated participating in more than 5 trainings per year.

Table 3 presents the descriptive statistics and the information entropy of the didactic-pedagogical indicators.

Regarding the importance attributed to the DPI, it was obtained that the general average of the indicators was 4.58. The lowest average (4.05) was calculated for the DPI3 indicator and the highest (4.80) was for the DPI14. As for the information entropy coefficient, the lowest was presented by the DPI7 indicator, with e(d 7) of 0.9989 and λ~i 0.0229 (2.29%); while the highest occurred in the DPI1 indicator, which obtained an e(d 1) of 0.9925 and a weight of 0.1544 (or 15.44%). These findings reveal that the DPI1 indicator showed the highest degree of dispersion between the scale levels, while the DPI7 had the lowest. This allows us to infer that most of the indicators did not present dispersion in the answers, that is, there is an alignment in the degree attributed by the respondents to the importance of the DPI.

Regarding the calculation of entropy related to execution, there was a general average of 4.29, with the lowest (3.49) being calculated in DPI3 and the highest (4.59) in DPI8. Regarding entropy, the lowest dispersion result obtained a weight of 2.84%, referring to DPI13, while the highest was calculated by DPI3 (15.89%). In addition to DPI3, DPI1 also showed a high dispersion coefficient (13.63%) when compared to the others, indicating that they had a higher degree of dispersion in the responses. Still, in relation to the indicators of this set, it is observed that the DPI2, DPI10 and DPI15 presented a median level of dispersion (6.99%, 6.79% and 7.63%) respectively, indicating little dispersion in the answers.

Table 4 highlights the entropy results of the evaluation indicators.

It is observed that the general average of the evaluation indicators was 4.57 for the level of importance and 4.11 for the level of execution. With regard to the level of importance, it is observed that the lowest mean (4.37) was found in EI3 and the highest (4.78) was in EI4. As for entropy, it is noted that EI3 was the only one to present a high coefficient of dispersion in the response scale with a weight of 41.63%, with the lowest average of the set (4.37). This was the indicator with the smallest number of respondents who attributed the maximum level of importance, implying that, while for some, it is essential, others evaluate it with little or no importance. EI4, on the other hand, obtained the highest average of the set and, consequently, presented the lowest degree of dispersion (weight 0.1168).

About the level of execution, it was found that the lowest average (3.94) corresponds to EI3, which showed a high level of dispersion in the response scale, as it showed a weight of 41.63%. In EI4, the degree of agreement between respondents was high, as in addition to obtaining the best average of the set, it presented low dispersion (11.68%), representing the lowest result of the group.

In Table 5, the findings regarding the entropy of the information of the indicators related to the questions of adopted methodologies are evidenced.

According to the data, the general average related to the importance of the indicators was 4.59. The lowest mean (4.34) corresponds to IMA4, while the highest (4.83) belongs to IMA3. IMA4 had the highest dispersion, compared to the other indicators in the set, with an entropy weight of 0.3414, and an average of 4.34, the lowest among the IMA.

Regarding the performance of the IMA, it is observed that the overall average was 4.25, with the lowest calculated by the IMA1 (3.93) and the highest calculated by the IMA3 (4.60). As for entropy, it is observed that the lowest result was for the IMA3 with a weight of 7.40%, indicating that there is no high degree of dispersion.

Next, Table 6 presents the results obtained for the entropy of the indicators related to the faculty.

The general average related to the importance of the indicators was 4.73, with the lowest (4.59) being for FI5 and the highest (4.80) for FI1. Regarding entropy, only FI4 and FI5 showed median levels of dispersion (weights 0.2027 and 0.2266) compared to the other indicators that make up a set, which showed similar consequences, indicating no entropy.

Concerning the performance of the indicators, the averages ranged from 4.12 (FI4) to 4.46 (FI6), resulting in an overall average of 4.31. Regarding the calculation of entropy, it is observed that three of the six indicators presented a degree of dispersion considered median in the responses. FI4 had the highest degree of dispersion in the responses, weighing 0.2476 (24.76%). To FI3, there is a median degree of dispersion (17.52), indicating that there is disagreement in the level of execution of this indicator in the courses represented in the research; however, to a lesser degree than FI2 and FI4, that is, the degree of agreement is greater than that of disagreement.

Next, the results of the infrastructure indicators are presented.

Regarding importance, the overall average of the indicators was 4.78, ranging from (4.75) for II2 and II5 to (4.84) for II3. As with the indicators related to the faculty, none with a high degree of dispersion were identified, only two median ones, the II2 with a weight of 24.03% and the II5 with a weight of 26.38%.

Regarding the execution, it was identified that the II presented the smallest variation among themselves regarding the levels of dispersion and the averages of the indicators. An overall average of 4.45 was calculated, with the lowest (4.40) being calculated in II5 and the highest (4.58) in II1.

Concerning the calculation of entropy, it was identified that the II5 was the only one to present a degree of dispersion between the response scale (25.91%), representing some level of disagreement among the respondents. As for the others, there is no dispersion level, since their weights varied from 0.1724 (II4) to 0.2070 (II2).

After the identification of the levels of entropy of the indicators per set, the calculation of entropy between the sets of indicators was performed (Table 8).

With regard to importance, the general average of the sets of indicators was 4.65, with the lowest (4.57) being found in the set related to the EI questions and the highest (4.78) corresponding to the II set. As for entropy, the lowest coefficient was presented by the DPI set, with e(d₁) of 0.9991 and λ~i 0.1464 (14.64%). On the other hand, the highest degree occurred in the EI questions, which obtained an e(d₂) of 0.9985 and a weight of 24.69%.

The IMA set also showed a higher level of dispersion to the other sets, with a weight of 22.33%. However, when analyzed together, it is observed that the EI and IMA sets did not present results with high degrees of dispersion, as they have similar weights to the others, indicating a low level of dispersion.

For the performance indicators, it was found that the overall average of the sets was 4.28, with the lowest (4.11) being calculated in the EI set and the highest (4.45) in the II set. As for entropy, the lowest result was presented by the DPI set, with e(d₁) of 0.9981 and λ~i 0.1302 (ie, 13.02%). On the other hand, the highest degree occurred in the IMA questions, with a low level of dispersion in the answers and a weight of 22.08%.

With regard to the level of importance, it is observed that most of the indicators showed a good level of agreement between the responses, however in all sets indicators with some degree of dispersion were calculated. These findings suggest that, despite the importance averages having surpassed level 4 of the response scale, for some respondents these indicators are very important, while others evaluate them as unimportant or very unimportant.

The same occurred in relation to execution, which in general, most indicators showed a good level of agreement, however, all sets presented indicators with some degree of dispersion. These findings reveal that those who presented dispersion results have a higher degree of disagreement in the answers, that is, while some perform it very frequently, there is a relatively proportional number who perform it less frequently or do not perform it at all.

4.2 Discussions

Among the observed results, it was found that all the proposed indicators have an attribution of greater importance than execution in the responses of the coordinators; that is, although the respondents assess that it is important to carry out such actions and strategies, they do not do it in the same way. Several reasons can cause this situation, such as the institutional policy or even the coordinator's lack of knowledge due to the short time in the function (^{ZANIN, 2014}; ^{MOLOZZI, 2015}). It is concluded that there is a convergence for the importance given by the coordinators to the curricular contents that allow the development of the professional profile of the egress (updated contents, adequate bibliographies, etc.). Still, on the other hand, there is disagreement about the importance of discussing the CPP of the course with first-year students. In addition, the coordinators converge on the execution of practical teaching activities. However, they have yet to focus so much on the discussion of the CPP of the course with the graduating students. These findings are out of line with what was highlighted by ^{Soares (2013)} when explaining that the role of the educational manager is premised on uniting the interests of students with the pedagogical project, noting the need of the market in relation to graduates of their institution.

Regarding the evaluation indicators, it is noted that the coordinators have a more significant divergence in the importance of carrying out evaluations in the Enade mock format. On the other hand, the coordinators have focused on encouraging teachers to give feedback to students based on the assessments carried out. The faculty indicator showed an excellent dispersion of coordinators regarding the importance of observing the faculty's work regime and prioritizing their professional experience.

Finally, it is concluded that the most critical evaluation indicators and which are most performed by the coordinators, involve the didactic-pedagogical, faculty, and infrastructure, which both in terms of importance and in terms of execution, present less dispersion of responses among the coordinators of the course of graduation. This is in line with what the legislation (Law 10.861/2004) currently recommends is to verify the teaching conditions offered to students in these three dimensions mentioned (^{BRASIL, 2004}).