1 Introduction

The assessment process of Higher Education Institutions (HEI) and Post Graduate Courses (PGC) is a fundamental mechanism that generates indicators for both control bodies and course coordinators to optimise their practices to offer high-performance professionals to the academia or the job market, as well as diagnose the aspects that must be improved. In this perspective, the qualification of PGCs becomes a decisive factor to enable the improvement of educational systems (Astakhova et al., 2016).

It is the universities’ responsibility to develop and encourage the employability of their students. In this perspective, universities must show society that they can train students with qualities useful to the job market through quality Education, in addition to citizen training (Caballero; Alvarez-González; López-Miguens, 2021; ^{Leal, 2023}). Among their purposes, ^{Bispo and Costa (2016)}, add that stricto sensu graduate courses train high-level researchers in a given area, professionals capable of working in Higher-Education teaching, and high-performance professionals through professional master’s courses.

Aiming at a greater efficiency in the teaching and learning process of the graduate courses, and the qualification of the courses offered, their assessment is necessary. In Brazil, this process is done through the Brazilian graduate assessment system. Among other goals and based on established criteria, the Coordination of Improvement of Higher Education Personnel (Capes) assesses existing courses and approves the proposal of new courses (Marques; Veiga; Borges, 2020). Capes has been undergoing constant updates encompassing technological, scientific and educational advances and included self-assessment as one of the evaluation criteria for graduate courses in 2018. Self-assessment is a way of self-knowledge of what happens in a course or institution, and allows an evaluation of the present situation of the course and the expectations. Such assessment also permits the evaluation of what has already been accomplished, how the administration takes place, what information is available for analysis and interpretation, and what the strengths and weaknesses of the institution are (^{Zainko; Pinto, 2008}). Self-assessment favours the construction of identity, heterogeneity and involvement of the courses assessed, in addition to the minimum standards guaranteed by external assessment (^{CAPES, 2019}). Also, ^{Seifert and Feliks (2018)} claim that self-assessment establishes what is being built, considering the teaching-learning process.

Meeting the CAPES assessment criteria, which include issues related to the dissertation and theses development process; course infrastructure; professors’ performance; curriculum structure, among others, this work aims to develop a Self-Assessment model of Graduate Courses from the student’s perspective.

The study presents an innovative bias for providing new evidence for the emerging literature on self-assessment applied to the graduate courses scenario, since the inclusion of this dimension in the Capes assessment system is recent and highlights a new aspect in the courses assessment field. It also innovates for being the pioneer on building a model that considers the aspects of self-assessment and for dealing with an empirical approach that can be replicated to different realities of the various graduate courses.

In addition, understanding the relationship between the different aspects of self-assessment and identifying the influencers in the assessment process and in the satisfaction of students and alumni is essential for graduate courses to achieve better results, as self-assessment provides relevant and updated information (Angst; Alves, 2018). Furthermore, the study can be useful to identify the central points to be addressed in institutional strategies aimed at improving graduate courses at HEI.

2 Model development

The development of a nation is directly related to the Education quality (Astakhova et al., 2016). Permanent assessment and continuous development of the educational context are necessary to meet social demands, measure quality and provide visibility to educational institutions. The assessment process with organised information favours the understanding of situations and relationships, the construction of meanings and knowledge about subjects, structures and activities that occur in an educational institution at a given time (^{Leite, 2008}).

In Brazil, the assessment is carried out every four years, seeking to analyse and assess the permanence of the PGCs offer (^{CAPES, 2014}). That evaluation contemplates three central dimensions. First, regarding the Program, the assessment seeks to check the operation, structure and planning of the Postgraduate Program (PPG) considering its characteristics and objectives. Second, the Educational Background considers the quality of the students, the performance of professors and the production of knowledge associated with the research and training activities of the program. Third the Impact on Society verifies the innovativeness of the intellectual production, internationalization, impact and social relevance of the program (^{BRASIL, 2019}).

In the last reformulation, implemented for the 2017–2020 quadrennium, the self-assessment of the graduate course started to be recommended and assessed by Capes. Self-assessment is present in the “Course” query, specifically in the “The processes, procedures and results of the course self-assessment, focusing on student training and intellectual production” item (CAPES, 2018).

Self-assessment is characterised by a process of self-reflection and self-analysis that highlights strengths and challenges that remain, aiming to improve the quality of academic work, training concepts, development and learning. Self-assessment is planned, executed and explored by the individuals who are responsible for the actions being assessed. It also allows for reflections on the context and policies applied, as well as the organisation of data that result in decision-making (^{CAPES, 2019}).

The self-assessment process must be broad both in its assessment dimensions and in the assumption of the participation of all those involved (^{Angst, 2017}; ^{Myalkina, 2019}). When creating a model, one must consider the need to cover all the activities of academic institutions, teaching, research, extension and management in a coordinated way, as well as crediting and supporting the various interested parties for the information (Monticelli et al., 2021).

Therefore, considering the need to implement effective self-assessment systems to meet Capes’s requirements, the internal assessment systems of educational institutions, and the multidimensionality necessary for a self-assessment model, this study proposes a Student Self-Assessment model with seven main dimensions: “HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support”, “Administrative Support”, “Course Assessment” and “Satisfaction with the Course”, which are related as shown in Figure 1 .

Source: Prepared by the authors (2022)

Figure 1 Student Self-Assessment Model 

The “course assessment” represents the graduate student’s self-assessment. This dimension is formed by three first-order constructs: the “Graduate Course Impacts”, “Professional Development” and “Selection Processes”. The “Graduate Course Impacts” dimension assesses the course performance in terms of teaching capacity, self-management and regional and national impacts, in addition to international performance. The “Professional Development” dimension analyses both the ability of the course to provide specific knowledge that effectively contributes to career development and to build researchers capable of producing nationally and internationally. On the other hand, the “Selection Process” dimension refers to the form, clarity and adequacy of the processes for selecting students and distributing scholarships (if any).

As determinants of the student’s self-assessment, five fundamental factors are considered: HEI Infrastructure, Course Subjects, Course Professors, Advisor Support, and Administrative Support. The first three factors are assessment parameters provided for in the assessment model of graduate courses used by Capes. The last two, which refer to the presence of support for students, are important support dimensions for the conclusion of a stricto sensu graduate course.

The “HEI Infrastructure” factor identifies the student’s perception of the adequacy of physical facilities, equipment, Internet access and security provided by the educational institution. The “Course Subjects” dimension comprehensively assesses the student’s perception of the adequacy of contents, bibliographies, activities developed, evaluation forms, and the quality of classes. In “Course Professors”, the student’s perception is sought both in terms of mastery of knowledge, teaching learning strategies and the form of service and access to the subject’s professors. In this sense, it is considered that in Higher Education, teaching-learning strategies generally require students to act more independently and proactively, with feedback being essential (^{Leal, 2023}).

In the support-related dimensions, both the course secretariat and the advisors are assessed. For the secretariat, the focus is on the adequate provision of information, accessibility, courtesy, and willingness to help students. The insertion of the “Advisor Support” dimension is justified by the evidence that the advisor-student relationship has a relevant role in retaining students in courses, in addition to being important in the knowledge construction (^{Moore, 2017}; Vieira et al. , 2022), since it helps students to fulfil their academic, personal and career goals (^{Martinez; Elue, 2020}; Meurer et al ., 2020). In the student’s perception, the dimension identifies the various facets of the guidance process, ranging from issues related to knowledge sharing to issues of access and feedback from the advisor.

The last dimension of the model is “Satisfaction with the Graduate Course”, which is directly impacted by the course assessment. This dimension represents the student’s level of satisfaction with the course in a comprehensive way, considering the main assessment parameters of a course according to the Capes assessment system.

Therefore, the Student Self-Assessment model is multidimensional, with seven central dimensions and three constructs that form the course assessment dimension. Of the seven dimensions, five are considered assessment antecedents (“HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support”, “Administrative Support”), and “Satisfaction” is the consequent dimension, that is, the one that is directly impacted by the course assessment. Table 1 presents the set of items that measure each of the model dimensions and their respective scales.

It is also noteworthy that internationalization, a challenge institutionalized both by national public policies and by institutional policies (^{Heinzle; Pereira, 2023}), depends on different actors to be executed (Vázquez; Jiménez García, Canan, 2022). The internationalization creates new opportunities, hastens the implementation of innovative work methods, improves mutual understanding between peoples and cultures, and contributes to the Education of a new generation for work in the global labor Market (Akkari et al ., 2023) was incorporated into the model based on specific items in different dimensions. Questions related to social impact were also incorporated into the model, which are represented by activities with the potential to highlight the link and impact of programs with society, in a virtuous circle between practice, research and professional qualification (^{Ferraço; Farias, 2021}).

3 Method

The present study is characterised as descriptive, carried out through a quantitative approach, from a survey-type research. According to ^{Malhotra (2019)}, the survey method is based on an interrogation of the participants carried out through a structured questionnaire, with the purpose of eliciting specific information from the interviewees.

The research was carried out with regularly enrolled students and alumni (since 2017) of graduate courses at the Federal University of Santa Maria (UFSM), constituting the population of this study (4,544 students and 7,970 alumni). The entire population received an invitation email from the institution, which resulted in a sample of 2,037 respondents.

As a collection tool, a questionnaire with 85 questions was used and organised into seven blocks. In the first block (“Graduate Courses Infrastructure”), a five-point adequacy scale was used (1– Totally inadequate; 5 – Totally adequate). The other blocks addressed issues related to the professors’ academic and professional quality, the advisor’s relationship and accessibility, the support from the secretariat, the general assessment of the graduate course, and the satisfaction with the graduate course. To measure the responses of these blocks, a five-point Likert scale was used (1 – Totally disagree; 5 – Totally agree). Finally, the last block was intended to characterise the respondents’ profile and collect some general information, with questions about gender, age, marital status, relationship, performance and length of service in the graduate courses.

In the first phase of the quantitative stage, an Exploratory Factor Analysis (EFA) was carried out to validate the proposed constructs using version 23 of the SPSS software (Statistical Package for the Social Sciences). Internal consistency was assessed by calculating Cronbach’s Alpha. As the analysis is exploratory, values equal to or greater than 0,7 were considered acceptable (Hair Junior et al ., 2014). In the second phase of the quantitative stage, Structural Equation Modelling (SEM) was used in order to validate the first-order constructs: “HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support”, “Administrative Support”, “Satisfaction with the Graduate Course”, and the second-order construct “Course Assessment”. The models were estimated using the Amos Graphics software.

To validate the constructs, the statistical significance of the indices was analysed: Chi-square (χ ² ), Chi-square/degrees of freedom ratio, Comparative Fit Index (CFI), Normed Fit Index (NFI), Tucker-Lewis Index (TLI), Root Mean Square Residual (RMR), Root Mean Square Error of Approximation (RMSEA). For the chi-square/degrees of freedom ratio, the recommendations are for values less than five. For CFI and TLI, values greater than 0,950 are suggested, and the RMR and RMSEA should be below 0,080 and 0,060, respectively (^{Byrne; 2010}; Hair Junior et al. , 2014; Hooper; Coughlan; Mullen; 2008; ^{Kline; 2015}).

As for the reliability measures, the coefficients of the Average Variance Extracted (AVE), Composite Reliability and Cronbach’s Alpha were considered. For the AVE, values equal to or greater than 0,5 are desirable (^{Fornell; Larcker, 1981}), with values from 0,7 admitted for the Composite Reliability and Cronbach’s Alpha (Hair Junior et al. , 2014). Discriminant validity was assessed in the nine constructs, comparing the AVE and the correlations. For this, it was necessary to consider the premises (^{Fornell; Larcker, 1981}), in which the correlation of all pairs of the model constructs was verified, comparing it with the square root of the AVE and the orientations of Bagozzi, Yi and Phillips (1991) through the Chi-square difference test. Finally, the estimation of the integrated model occurred through the SEM. In this step, the chi-square indices (χ ² ), the Chi-square/degrees of freedom ratio, CFI, NFI, TLI, RMR and RMSEA were considered in the validation of the integrated model.

4 Results

In order to identify the self-assessment dimensions of the PGCs, and the satisfaction and general assessment of the courses, the EFA was applied, with the varimax rotation method and as an extraction criterion, eigenvalues greater than one were defined. Also, all initial variables showed a commonality greater than 0,5 and were maintained in the analysis. The exploratory factor results of the analyses are presented in Table 2 .

Bartlett’s test rejected the null hypothesis that the data matrix is an identity matrix for all dimensions, indicating the data factorability. The factor loadings of the variables range from 0,405 to 0,924, demonstrating that some variables have a greater contribution to the item’s formation. Cronbach’s alphas are adequate, as they all resulted above 0,7. Therefore, the EFA confirms the dimensions predicted in the theoretical model and their respective items.

4.2 Integrated model analysis

To analyse the integrated model, the first procedure performed was the validation of the second-order construct, called “Course Assessment”. The results of the validation procedure are shown in Table 5 .

Table 5 Adjustment indices of the second-order construct and the integrated model 

Indices	Criteria	Course Assessment (second-order construct)		Integrated Model
		IM	FM	IM	FM
x 2 (value)	---	162,246	367,400	17465,421	6104,581
x 2 (probability)	>0,05	0,000	0,000	0,000	0,000
x 2 /degrees of freedom	< 5	28,443	7,654	6,687	2,769
CFI – Comparative Fit Index	> 0,95	0,889	0,977	0,885	0,966
TLI – Tucker-Lewis Index	> 0,95	0,847	0,963	0,878	0,963
RMSR – Root Mean Square Residual	< 0,08	0,069	0,032	0,250	0,044
RMSEA – R.M.S. Error of Approximation	< 0,06	0,116	0,057	0,053	0,029

Source: Prepared by the authors (2022)

Once the second-order construct was validated, the analysis of the integrated model was performed. To verify the validity of the integrated model, analyses of the Chi-square, Chi-square/degrees of freedom, CFI, TLI, RMSR and RMSEA indices were used. The results are also presented in Table 5 .

The proposed model initially presented inadequate adjustment indices. Therefore, the strategy of including correlations between the variables errors that made theoretical sense was adopted. Thus, Figure 2 presents the final integrated model with standardised coefficients and significance of the relationships between the constructs.

Source: Prepared by the authors (2022)

Figure 2 Final Model 

The relationship evidenced in the theoretical model shown in Figure 2 demonstrates that the assessment of graduate courses is directly influenced by the “HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support” and “Administrative Support” factors. The highest impact is exerted by the course subjects (0,525) and the course professors (0,200). This is in line with studies carried out by De Rezende et al. (2017), who found that the faculty also directly impacts the academic performance of graduate students. They are also consistent with the literature that highlights the role of the teacher by promoting a harmonious and democratic dialogue with students with respect, availability, care and receptivity, (Alves; Espindola; Bianchetti, 2012) and so, directly contributes to a positive self-evaluation of the course.

It can also be seen that the course assessment is formed by the first-order constructs: “Graduate Course Impacts”, “Professional Development” and “Selection Processes”, which have a regression load higher than 0,8, indicating a strong influence on the second-order construct.

Another point to be highlighted from the model results is that the course assessment has a significant influence on the course satisfaction. In other words, there is a causal relationship between two constructs so that better course evaluation results imply a better perception of satisfaction by students and alumni.

It is also necessary to highlight that the variables defended in the projection of the initial model were, for the most part, accepted after the estimations and validity tests, corroborating the adequacy of the proposed model. The results of the factor loadings of each item (observed variable) in relation to its first-order construct are shown in Appendix A.

Finally, the averages were estimated to identify the perception of students and alumni in each of the model dimensions of the model.

For the five antecedents of the course assessment, there were four averages: “HEI Infrastructure” (3,63), “Course Subjects” (4,21), “Course Professors” (4,23), “Advisor Support” (4,35) and “Administrative Support” (4,52). Considering that the scale varies from 1 to 5, it appears that the respondents consider the “HEI Infrastructure” very adequate and that they agree with most of the issues related to Subjects, Professors and Support. Also, for these three constructs: “Graduate Course Impacts” (4,26), “Professional Development” (4,35) and “Selection Processes” (4,32), and for the “Course Assessment” factor (4,31), the averages are slightly higher than 4, stating that students and alumni positively assess the graduate course. Furthermore, it is observed that the “Satisfaction with the Graduate Course” is high, as it reached an average of 8,106 on a scale from zero (totally unsatisfied) to ten (totally satisfied). Therefore, the average results indicate that the institution’s graduate courses are well-assessed, and that students and alumni are satisfied.

Thus, the estimation and validation results presented in this section demonstrate the possibility of using a very comprehensive and complete model for student self-assessment. In line with the self-assessment literature that suggests the need to create models capable of highlighting the complexity, plurality and challenges of self-assessment in university institutions, since it is influenced by structural and situational elements (Lehfeld et al., 2010). Such results are essential because they are associated with the development of students’ skills and their employability (ST Jorre; Oliver, 2018).

5 Final Considerations

Self-assessment as an assessment process has several dimensions and encompasses the perception of different stakeholders, that is, it includes professors, students, the community, and alumni. In this study, the main actor is the student, and the main goal is their self-assessment of the graduate course.

For this, a multidimensional model of self-assessment was developed with five evaluation antecedents (“HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support”, “Administrative Support”) and “Satisfaction” is the dimension directly impacted by the course assessment.

In the process of building the model, several validation tests were performed, which analysed and indicated the unidimensionality and the convergent and discriminant validity of all the proposed dimensions. The estimation of the integrated model confirmed the impact of antecedents on the graduate course assessment and its direct influence on satisfaction. Altogether, these results indicated that the proposed self-assessment model is robust and parsimonious.

Since Capes has included self-assessment as one of the assessments of graduate courses, the use of student self-assessment models should be adopted by all programs, so the self-assessment model presented in this article can become useful to several courses. The use of a multidimensional measure allows a broad approach to the different aspects that impact directly on the course assessment and indirectly on the satisfaction of students and alumni. It is also worth noting that the measure as a whole or some of its dimensions can be used to assess other stakeholders in the self-assessment process, such as alumni.

The practice of student self-assessment, by itself, is already a fundamental process for self-regulated and lifelong learning (^{Yan, 2020}), as it leads students to reflect on the gap between current and desirable performance levels (^{Yan; Carless, 2022}). However, Brazil still does not have a culture favorable to the self-assessment process, which is often criticized for inducing behaviors and valuations, either positive and negative, based on indicators that are not always accepted or seen as the most appropriate (Leite et al., 2020).

Thus, the construction of adequate measurement models is just one of the first conditions for the implementation of a self-assessment policy capable of effectively indicating the strengths and possible improvements to graduate courses managers. Just with the development of a continuous self-assessment policy that promotes effective feedback a country’s graduate system evolves. Thus, in addition to choosing an appropriate model, it is necessary for graduate courses to establish their own self-assessment policy, taking into account both the rules established by Capes and their own criteria.

As for the research limitations, we can mention the application of the questionnaire via the Internet and the fact that the research was carried out only at one university. It is suggested that future works use other samples and analyse the possible differences arising from the course level (master’s and doctorate). New scale validation tests can also be performed.