Introduction
The quality of education is important for the development of a country and can be considered as a form of public asset (SANTOS, 2011). Therefore, the pursuit for quality in education has been the object of study within several research fields, with the field of institutional assessment currently gaining strength, emphasizing the challenges that involve the diagnosis of the inner workings of an educational institution (BALDIGEN, 2018).
In Brazil, the National Higher Education Assessment System ( Sistema Nacional de Avaliação da Educação Superior – SINAES) was created to improve the quality of higher education; guide its expansion; permanently increase its institutional, academic, and social effectiveness; and, especially, to promote the deepening of the social commitments and responsibilities of higher education institutions (HEI). For the evaluation of the institutions, diversified procedures and instruments are used, including self-assessment and external evaluation in loco (BRASIL, 2004). The former is the focus of this study.
Institutional self-assessment (ISA), a component of SINAES, is a continuous process in which the organization builds knowledge on its own reality, seeking to understand the meanings of its activities to achieve greater social relevance. To this end, the evaluation systematizes information; collectively analyzes the meanings of its achievements; unravels forms of organization, management, and action; identifies weaknesses, as well as strengths and potential; and establishes strategies to overcome problems. Such assessments are legally required to be conducted by the HEI. Based on the principle of university autonomy, each institution is responsible for developing the process in the manner most appropriate to its context. The evaluation has a formative character and will constitute the basic framework for the regulation and supervision processes within higher education, thus promoting the improvement of its quality.
Considering the role of the self-assessment processes within the context of higher education regulations, we observed that many HEI perform the assessment only to meet the requirements of regulatory bodies, instead of using it as a management and planning instrument, making the self-assessment process a bureaucratic issue (LIMA, C. I., 2010) rather than an institutional management policy. This process offers the potential to manage transformations since it can be used to support managers’ decision-making process ( NARDELLI, 2019; GONÇALVES FILHO, 2016). Universities, however, have difficulty in using self-assessment as a management tool ( CARVALHO; OLIVEIRA; LIMA, 2018; AOKI, 2017; ROSA et al., 2011 ) the step by step of this process and the effective use of its results are not presented in the literature. Additionally, this study presents research gaps, seeking challenging opportunities for researchers to establish new research flows and for managers and decision-makers to formulate effective implementation strategies. Thus, we seek to answer the following research question: What guidelines are used for self-assessment in higher education institutions? Identifying the current attributes of this process may assist in the development of better practices in this area.
To systematize elements that may assist in solving the research problem, we sought to analyze the guidelines for self- assessment methods in higher education institutions, considering their processes, results, and effectiveness. Guidelines are instructions that support the development of an institutional self-assessment systematicapp which considers the processes and results for effective decision-making in university management.
Self-assessment in higher education institutions
Evaluations became important as a management tool in social organizations, mainly with the publication on the principles of scientific administration by Frederic Taylor, in 1911 ( Figure 1). This work criticizes administrators for making decisions based on intuitions and experiences, without using performance standards.
There is not a single definition for evaluation. The classic concept – proposed, and later complemented, by Scriven ( SCRIVEN, 1967, 1994) – states that evaluation is the judgment of importance, merit, and value. The evaluation must be contextualized, conversing with the object to be evaluated, its theoretical assumptions, its political articulations, and the actors involved (ARRUDA; PASCHOAL; DEMO, 2019). Evaluating is not just about raising numbers, but raising actions based on those numbers. The assessment must be credible, but it cannot claim to be the conclusive demonstration of the truth. Its function is not to reveal nor determine the truth, but rather to ground the possibilities for reflective social processes that produce meanings on actions, relationships, and educational productions (DIAS SOBRINHO, 2008).
According to Souza (2010), the Organization for Economic Cooperation and Development (OECD) was the only international institution in the 1960s that evaluated educational systems worldwide, providing information for the formulation and monitoring of policies for socioeconomic development. The OECD has been operating since then (OECD, 2011), and they annually present an overview of world education within its member countries, Brazil included.
The first discussions about institutional evaluation in Brazil started in the 1980s (BARREYRO; ROTHEN, 2006). Brazilian public universities argued that the evaluation was necessary so that the principle of transparency could be fulfilled, that is, of accountability to society. During this period, the discussion on evaluation gained another perspective: not only was it an instrument of transparency but also a concern with quality and autonomy (BALZAN; DIAS SOBRINHO, 1995). Since the early 1990s, the theme of institutional evaluation has been gaining consistency regarding the principles of university autonomy and quality. Several attempts to evaluate higher education institutions have been made. In 2004, however, institutional self-assessment became mandatory in Brazil, subject to the National Higher Education Evaluation System, with the aim of ensuring the national evaluation process in higher education institutions.
National higher education assessment system
Established by Law No. 10,861 of 2004, SINAES aimed to improve the quality of higher education; guide the expansion of its offer; permanently increase its institutional, academic, and social effectiveness; and, especially, promote the deepening of social commitments and responsibilities of higher education institutions, by promoting their public mission, their democratic values, the respect for difference and diversity, the affirmation of autonomy, and their institutional identity (BRASIL, 2004).
The evaluation of higher education institutions aims to identify their profile and the meaning of their performance by their activities, courses, programs, projects, and sectors, considering different institutional dimensions, such as: I – mission and institutional development plan; II – the policy for teaching, research, and extension; III – the social responsibility of the institution; IV – communication with society; V – personnel policies; VI – organization and management of the institution; VII – physical infrastructure; VIII – planning and evaluation, especially the processes, results, and effectiveness of institutional self-evaluations; IX – student service policies; and X – financial sustainability. For the assessment of institutions, diversified procedures and instruments are used, among which is the self-assessment (BRASIL, 2004).
Each university develops the best way to conduct its self-assessment process. This study is developed from the perspective of institutional self-assessment and, in section 4.2, we show the systemic analysis for the dimension of planning and evaluation. The ISA contains analyses, criticisms, and suggestions and is configured as an important instrument for decision-making insofar as it conceives a collective discussion about the institution with the subjects that compose it, giving legitimacy, autonomy, and meaning to this analysis. This process enables transparency and, thus, allows for decision-making that makes public spending more efficient (ARRUDA; PASCHOAL; DEMO, 2019). However, the scenario identified by Cunha (2010) reveals that the use of ISA results represents a challenge for Brazilian HEI, contributing little to university management. According to Arruda, Paschoal, and Demo (2019), the strongest criticism made of the evaluation is that its results are not used for decision-making, which raises the question of the relevance of the evaluation itself.
Methodological procedures
This subsection presents the methodological framework of the investigation to inform the assumptions that guided its execution. From the point of view of nature, this qualitative and quantitative research is classified as basic since it aims to generate new knowledge useful for the advancement of science without an expected practical application. It has an exploratory-descriptive character: it is exploratory since the main agent is the researcher who generates knowledge on self-assessment in higher education institutions with the purpose of providing more information on the subject to be investigated; it is descriptive since the researcher only records and describes the facts observed without interfering with them (PRODANOV; FREITAS, 2013).
The technical procedures, that is, the way in which the necessary data were obtained for the elaboration of the research, was elaborated from material already published, consisting of scientific articles and national theses and dissertations. Regarding data collection, both primary and secondary data were used (PRODANOV; FREITAS, 2013). The stage of selection of the bibliographic portfolio (BP) made use of primary and secondary data.
ProKnow-C (ENSSLIN; ENSLLIN; DUTRA, 2015), the instrument used to assist in the operationalization of this research, is composed of four phases ( Figure 2).
The selection process of the bibliographic portfolio, the first stage of ProKnow-C, begins with the definition of research axes and their keywords ( Figure 3). For searching the databases, keywords in English were used, in addition to their respective translations and adaptations to the Portuguese language. Our study focuses on studies on institutional self-assessments in higher education institutions.
The databases selected for this research were Scopus, Web of Science, EBSCOhost, Engineering Village (Compendex), and SciELO since they are those of the highest priority in the area. The Brazilian Digital Library of Theses and Dissertations (BDTD) was also searched, since this is a topic with specificities in Brazil, in addition to searches conducted on Google and Google Scholar. There was no time limitation in any of the filter searches and the search was carried out looking for the defined keywords within titles, abstract, and/or keywords of the sources. The search was limited to articles published in English and Portuguese. However, there was no distinction in the type of article, that is, articles from both journals and congresses were considered. The database search resulted in 608 documents, totaling 327 after duplicates were eliminated.
A total of 970 theses and dissertations were obtained, 279 publications were identified as duplicates by EndNote X9 and were subsequently excluded, resulting in 328 references. In the filtering phase of the raw article bank, the title of each article was examined and those that were clearly misaligned were excluded for not contributing to this research, according to our pre-defined criteria (self-assessment and higher education institution). The search on the databases resulted in 196 documents with eligible titles. These same criteria were used for the other sources of this research, resulting in 106 (33 theses and 73 dissertations) documents with eligible titles in the BDTD. The documents with eligible titles found at Google and Google Scholar totaled 39. Three of these were duplicated from the databases and six derived from BDTD (three theses and three dissertations).
After completing the next step – which aimed to identify the alignment of the abstracts with the author’s interest (presenting characteristics of the self-assessment processes in higher education institutions) – 44 documents were obtained from the databases; 23 theses; 37 dissertations and; nine documents from Google and Google Scholar; eight articles (one from a congress and seven from journals); and one thesis. For organizational reasons to assist in the analysis, these last two were added to the database and BDTD and resulted in 113 documents.
The references of the theses and dissertations and articles with more than 10 citations were analyzed, since these comprised 85% of the total citations of the articles already selected. This process was an opportunity to check if there was any relevant work according to the same pre-established criteria. Six new documents were found (a thesis, a dissertation, and four articles), totaling 119 documents which make up the study portfolio of this article.
Based on this portfolio on self-assessment in higher education institutions, 62 documents described some forms of guidelines that considered planning and management for decision-making, these were the basis for the results of the bibliometric and systemic analysis of the portfolio.
Presentation and discussion of results
The bibliometric analysis of the results, the second stage of ProKnow-C, aims to highlight the variables identified in the bibliographic portfolio. From this information, complementary data is searched and analyzed to build knowledge on the subject. This procedure is performed by counting the occurrence of a certain variable within the bibliographic portfolio, as well as in its references. This section also presents the systemic analysis of the results and, finally, presents research opportunities.
Bibliometric analysis
The bibliometric analysis of the results, considering the basic variables of the articles from the databases, shows: (1) number of articles per year of publication; and (2) production by country in which the study was developed. In relation to theses and dissertations, it exposes: (3) quantity of publication of theses and dissertations per year; and (4) concentration area of the author’s graduate program.
Figure 4 corroborates Balzan and Dias Sobrinho (1995), since it points to the first discussions about institutional evaluation in the early 1980s, although more articles have been published in the last five years.
Figure 5 shows that Brazil was the country with the most articles published on the selected subject, followed by the United States of America and Argentina. The other countries had a small number of studies carried out on the subject.
From the BDTD documents – theses and dissertations – an increase in publications in the last decade is noticeable ( Figure 6).
More than half of the theses and dissertations in the bibliographic portfolio were from the area of Education, followed by Production Engineering and Administration, respectively ( Figure 7).
Systemic analysis
The systemic analysis of the results, the third stage of ProKnow-C, consists in analyzing the content and verifying the characteristics of the bibliographic portfolio on the subject to identify knowledge gaps. From this stage on, the portfolio analysis becomes one, that is, the articles and the theses and dissertations are grouped together. All dimensions of SINAES must be considered in the institutional self-assessment process (RIBEIRO, J., 2015). Our study, however, is limited to the analysis of the portfolio from the perspective of planning and assessment, especially the processes, results, and effectiveness of the self-assessment.
This decision is based on the importance of that dimension in the evaluation process in higher education institutions. The analysis from this perspective implies subsidies for university management, especially regarding decision- making. Thus, Table 1 points out guidelines for self-assessment in higher education institutions. Some guidelines were proposed in this system and corroborated in the bibliographic portfolio, others came from specific experiences of the authors of the 62 documents.
Source: Elaborated by the authors.
The planning phase – predominantly preceding the institutional self-assessment process – comprises the definition of the organization’s goals, the establishment of a strategy, and the development of an action plan to coordinate activities. Souza (2010) states that the assessment articulated with planning allows the university to deal better with reality. Nevertheless, Campos (2019) highlights the lack of integration of assessment with institutional objectives and planning. Therefore, the characteristics of the institution, its mission, vision, and objectives (GOULART, 2018), its size, and the existence or not of previous evaluative experiences (AOKI, 2017) must be considered. The assessment must enable the educational institution to meet the demand for higher education in the social environment in which it is inserted; it must respect the institutional identity, history, and culture, by using historical series of indicators (ANDRIOLA; ARAÚJO, 2016).
For the evaluation to be used as subsidy for decision-making processes, the results must be analyzed beyond measurement and classification, thus allowing the occurrence of judgment. Therefore, for the assessment to make sense, it needs to be transparent (ANGST; ALVES, 2018), interpreted, discussed, and incorporated by the academic community (NARDELLI, 2019), generating feedback on actions ( CARDOSO, 2017; ANDRADE, 2014).
The use of the results generated by the evaluation process offers conditions for decision-making that can lead to improvements in HEI. However, empirical data showed the lack of commitment to the evaluation process and the non- use of the results (BOTELHO, 2016). The reflection and the judgment of the results have been a complicated task to be performed, according to Carvalho, Oliveira, and Lima (2018). The results of the actions must encompass all those involved, making awareness crucial for this process. Thus, organizational management can continuously improve, as long as it considers the needs of those involved, showcasing the implementation of improvements, generating transformation.
Using the results from the institutional self-assessment process is effective due to several factors, including the creation of an evaluative culture (SAIS, 2017) in the academic community; satisfactory communication during the process (ARRUDA; PASCHOAL; DEMO, 2019); promotion of freedom of expression (ANDRIOLA; ARAÚJO, 2016); continuity so that these improvement practices are not isolated moments (BERNARDES; ROTHEN, 2016); and permission for a global and systemic view (ANGST; ALVES, 2018) of institutional actions.
The discourse of managers uncovers a conception of formative self-assessment and provides opportunities for institutional improvement from their results. The actions practiced by managers do not confirm the effectiveness of SINAES regarding the use of evaluation results to produce improvement actions that aim at increasing institutional quality. The bibliographic review showed the importance of self-assessment; the process, however, needs to mature in order to lead the promotion of the institution’s autonomy, in a reflective and transforming bias. Therefore, the result of this research corroborates portfolio studies that argues that the academic community needs to award new meaning to institutional self-assessment.
Research opportunities
In the fourth stage of ProKnow-C – analyses of the 62 documents with described guidelines – we perceived knowledge gaps within self-assessment in which new research flows can be established for the development of knowledge aimed at manager and decision-makers to help them formulate more effective implementation strategies. Along with the gaps, the guidelines for self-assessment identified in the literature can be considered when developing a systematic approach that aims to formalize processes and use results effectively. Table 2 shows possible research opportunities to address the identified gaps, which are considered in conjunction with the guidelines for the further development of a systematic approach.
Gaps | Opportunities | |
---|---|---|
PROCESS | - Lack of coordination between processes and agents involved | - Modeling of processes, in which the tasks, mode of operation, deadlines, and those responsible are evident |
- Lack of integration of ISA with institutional objectives and planning | - Integration of ISA with other institutional instruments, such as the IDP, allowing a systematic view of the institution | |
RESULTS | - Lack of reflection on the results | - Proposition of actions for those involved based on the analysis of results |
- Lack of qualitative results | - Use of indicators, not only quantitative, but also qualitative | |
- Lack of clarity in the process that leads to inadequate results | - Use of the principles of the new public management, such as transparency, objectivity, and clarity | |
- Punctual results, without monitoring improvements | - Insertion of a continuous flow of information in the work routine | |
EFFECTIVENESS | - Data obtained are often not essential in the demands of each academic segment | - Research on the expectations of each segment of the academic community |
- Insufficient impact of the evaluation on management | - Change of mental model, in which people see the value in the institutional self-assessment process | |
- Main regulatory aspects in ISA |
Source: Elaborated by the authors.
Although this issue has been widely discussed, in practice, little has been done regarding models or systematics of institutional self-assessment that present the processes in a formal way, with a presentation of tasks, mode of operation, deadlines, and of those responsible. Process modeling helps in planning and monitoring actions. We also observed a difficulty in creating an evaluative culture in which both those responsible for conducting ISA and the overall academic community use the process as a work routine in a cycle of continuous improvement, directly impacting the value of the service provided to society. Thus, this article offers opportunities to discuss future interventions in the ISA process and contributes to filling some of the gaps regarding the use of evaluation results, offering practical and theoretical-academic contributions.
Based on the opportunities described in Table 2 and considering the proposition of a model of institutional self- assessment based on processes, results, and effectiveness, we suggest future studies focus on the application of two approaches: new public management or lean management. New public management emerged in the 1990s (HOOD, 1991) and uses the principles of transparency, objectivity, and clarity in the process. As with other market segments, education requires customer focus, responsive management, and greater production with less cost. We also suggest the use of concepts derived from lean management in the development of a model of institutional self-assessment that aims to improve work and develop people. According to the Lean Institute Brasil (2016), this approach can help in the management of processes since it can systematically eliminate waste and solve problems in people’s day-to-day lives. Lean thinking prioritizes the overall mentality of the system, guiding those involved to look at the internal factors of the organization’s development, giving preference to the efficient use of resources, guiding the maximization of the results and the involvement of teams in search of continuous improvement.
Final remarks
We can observe a change in behavior in the literature; ideas are currently more focused on issues related to human resources and the performance of organizations, while, in the 1980s, the evaluation had a control function. This study highlights the need to go beyond evaluations and monitoring of performance management, especially in complex organizations like universities, in which academics, technicians, teachers, and managers have goals. Gathering the information is easy; the application of the results, however, is challenging for management. Thus, there is no pattern that can be replicated successfully in this system, it is necessary to fit the environment and reality in which the organization operates.
Thus, this study analyzed the literature guidelines for self-assessment methods in higher education institutions, regarding processes, results, and effectiveness. The analysis of attributes already proposed for this process helps in the development of a best practice in this area. The use of both the new public management and the lean management approaches can achieve successful results and is indicated in this study. The fact that much of the information on self-assessment with little practical value is available may be linked to the obligation that SINAES imposes on HEI, while higher education institutions are adhering to the principle of “voluntary adhesion,” that is, it is allowed for the academic community to participate in this process.
The information collected shows that, in Brazil, social and geographical differences are prevalent, resulting in differences among HEI. This study corroborates Andrade (2014); although the process of self-assessment in higher education has potential and can be taken as an indication of advancing a culture of self-assessment, not enough space is given to debate the results of the evaluations, much less to planning new actions. This study is expected to promote the self-assessment process and this will lead to constant improvement.
Discussions on the evaluation process and its results, by the internal academic community, is essential for the development of management policies and institutional planning. This is an uninterrupted quality search procedure and requires predisposition for transformation. Distancing the process from the culture of change is unconceivable. Therefore, it must be a constant object of reflection, making it an instrument that generates valid and necessary information that supports the academic community in the planning and management of activities. Self-assessment must be formative, continuous, and permanent. Moreover, with the current scenario of higher education, it should impose important changes in the methods of operation and management of universities.