INTRODUCTION

The past two decades have been especially fruitful in terms of goal setting and legislation change regarding the European higher education (HE). In the European context, in addition to globalization trends and challenges worldwide, the Bologna Process (BP) brought about a series of specific changes aimed at creating a European Higher Education Area (EHEA), thus promoting the comparability of the standards and quality of higher-education qualifications.

In Portugal, the Decreto-Lei [Decree-Law] n. 74 (2006) (later amended by Decreto-Lei n. 107, 2008) provided for the application of Bologna Principles in HE institutions. Formally, Portugal has followed structural changes in study cycles according to European standardized criteria, and has implemented a quality improvement system. The changes involve a transition from a system based on “knowledge transmission” to one based on “student competence development”, with an emphasis on experimental work, project work, and transversal skills (Decreto-Lei n. 107, 2008). The goal of these changes is to foster more competitive learning environments, as well as conceptual changes regarding teaching (^{Reimann & Wilson, 2012}), recognizing the central role of the student within a logic of autonomy, teamwork and active learning (^{Segers & Dochy, 2001}). The Bologna Process has implied change in various dimensions, especially concerning curriculum flexibility and teaching and student work (^{Pereira & Flores, 2016}).

Studies undertaken in the Portuguese HE context have shown the commitment of HE institutions to BP, as well as increased cooperation between Portugal and the European HE institutions (^{David & Abreu, 2017}). However, some contradictions were also identified, such as: the apparent destruction of the binary system (university and polytechnic education) and the complex preservation of the national culture, language, education systems and institutional autonomy (David & Abreu, 2007); the successful realization of more formal aspects of the BP (^{Veiga & Amaral, 2008}) and the maintenance of traditional assessment practices (^{Pires et al., 2013}); and the prevalence of practices not aligned with the Bologna purposes (Pires et al., 2013; European Students’ Union [ESU], 2015; ^{Barreira et al., 2015}; ^{Pereira et al., 2017}; European Commission, 2018). Other findings concern mixed assessment methods, including more traditional methods alongside more innovative practices (Pires et al., 2013; Pereira et al., 2017; ^{Flores & Pereira, 2019}).

Despite the positive results, adapting HE programs to the Bologna teaching and learning perspective is still a challenge. In some cases, new and innovative assessment practices have been introduced towards a more student-centered approach. However, in most cases, assessment is still about “making statements about students’ weaknesses and strengths” (^{Earl & Katz, 2006}, p. 4).

Traditionally, classroom assessment has focused on its summative function, on the assessment and measuring of learning, using assessment information to make and report judgements about learners’ performance which may encourage comparison and competition (^{Black & Wiliam, 1998}). On the other hand, the formative function of assessment, which refers to how student learning can be improved (Black & Wiliam, 1998), has also been highlighted.

Although more traditional forms of assessment may be effective in specific contexts and purposes, they are not suitable for all assessment purposes and may encourage reproduction and memorization (^{Perrenoud, 1999}; ^{Biggs, 2003}). In turn, the so-called student-centered assessment methods (^{Webber & Tschepikow, 2013}; ^{Myers & Myers, 2015}) enable the development of technical and problem-solving skills, e.g., fostering greater student involvement in the learning process (Myers & Myers, 2015). These methods imply more global tasks that have to be developed over time. They can simultaneously include the process and product and the individual and collective dimensions. In addition, they stimulate autonomy, collaboration, responsibility, constructive feedback, interaction with stakeholders and peers, knowledge building (Webber & Tschepikow, 2013), skills development, and the deepening of learning (^{Brew et al., 2009}). These methods include, for example, experimental hands-on work, project work or reflections (Webber & Tschepikow, 2013; ^{Struyven et al., 2005}).

Regardless of the assessment approach chosen, it is important to deal “with all aspects of assessment in an integrated way” (^{Black & Wiliam, 2018}, p. 552). The selection of assessment strategies represents a key component of the teaching and learning process. This selection should result in a goal-oriented questioning process in the program or course (^{Hadji, 2001}), defining which questions should be answered through the assessment. Therefore, it is essential to use a variety of assessment methods based on their suitability for teaching and learning goals and the nature of courses and curricular units (^{Earl & Katz, 2006}; Black & Wiliam, 2018).

An effective and successful questioning within the EHEA context implies proper goal formulation and the recognition of assessment as a cornerstone of the teaching and learning process. With the Bologna Process, there is an expectation of more diverse assessment methods (i.e., alternative methods, including those that involve students) and the adoption of other approaches to assessment, such as assessment for learning (^{McDowell et al., 2011}), recognizing the role of feedback in the assessment and learning process (^{Black & Wiliam, 1998}; ^{Hattie & Timperley, 2007}; ^{Carless et al., 2011}). Within the context of learner-centered practices, the literature also recognizes the role of classroom in the organization of innovative and facilitated learning environments (Black & Wiliam, 1998) and in the development of innovative ways of structuring teaching and assessment (^{Fernandes, 2015}).

This view has gained prominence in the context of the BP (^{Flores & Simão, 2007}), challenging faculty to promote more questioning, innovative and creative learning opportunities (^{Zabalza, 2008}). However, it is not clear whether faculty have fully embraced this (^{Webber & Tschepikow, 2013}).

This paper¹ reports on research aimed at understanding HE assessment practices after the BP, particularly concerning possible changes, and the meaning of these changes, taking into account the context of university teaching.

METHODS

This paper draws upon a broader Ph.D. research in educational sciences, funded by the Fundação para a Ciência e a Tecnologia [National Foundation for Science and Technology] (Ref. SFRH/BD/103291/2014) and which itself is a part of a broader three-year research project entitled “Assessment in higher education: The potential of alternative methods”, funded by Fundação Nacional para a Ciência e a Tecnologia [National Foundation for Science and Technology] (Ref. PTDC/MHC-CED/2703/2014). The study reported in this paper aimed to identify assessment practices of faculty and investigate the influence of the BP on those practices. In particular, the following research questions are addressed:

A survey was carried out in five Portuguese public universities, combining closed and open-ended questions.

ANALYSIS

Statistical analyses were performed using the software IBM SPSS Statistics 24. Factor analysis is a technique that can be used not only to understand the structure of a measure, but also to reduce data to a more manageable set of variables (factors or dimensions), while retaining as much original information as possible (^{Field, 2009}). Therefore, exploratory factor analysis (EFA) was performed, using the principal component analysis method with varimax rotation (Field, 2009). The assumptions of adequacy of the sample’s size and sphericity were verified. In this study, this analysis served two purposes: (a) to obtain empirical evidence for the underlying structure of the scale on assessment practices, by identifying core dimensions, based on the intercorrelations between the items; and (b) to reduce the data obtained through the 30 items to more manageable scores that are interpretable and represent the core dimensions of the assessment practices. The number of factors and items to be retained was based on two criteria: factors with eigenvalues greater than 1; and items with factor loadings greater than 0.35. The fidelity of the subscales was tested with Cronbach’s alpha. Because this is an exploratory study, alpha values greater than 0.60 were considered (^{Hair et al., 2009}).

The mean score of the items was calculated for each dimension, and the relationship between these dimensions and participants’ demographic and professional characteristics was explored. To investigate the effects of age, gender, years of experience, professional category, field of knowledge, cycles of studies taught, and pedagogical training on the assessment practices, multivariate analysis of variance (MANOVA) was used. The assumptions of independence of the observations, univariate normality and homogeneity of the variance-covariance matrices were guaranteed (^{Field, 2009}). The partial eta-squared (ηp²) values were calculated as a measure of the effect size, considering the following guidelines for its interpretation: small effect, ηp² > 0.01; medium effect, ηp² > 0.06; large effect, ηp² > 0.14 (Cohen, 1988). Regarding demographic characteristics, in the age variable for this sub-study, two groups were considered for analysis: 1) faculty under 45 years of age; 2) faculty over 45 years of age. In the variable professional experience (professional variables), the length of service was added as follows: 1) faculty with less than 15 years of experience; 2) faculty with more than 15 years of experience. The cut-off points for the grouping in the variables “age” and “professional experience” were defined based on the date of implementation of the Bologna Process in Portugal. Therefore, older faculty and faculty with more years of experience most likely had experience in the two periods (before and after the Bologna Process), whereas the younger and less experienced faculty most likely did not. The professional categories “associate professor with habilitation” and “assistant professor with habilitation”, and “associate professor” and “assistant professor”, and the fields of knowledge of exact sciences and engineering and technology sciences were also grouped in order to obtain more adequate units of analysis.

Data collected through the open-ended questions of the questionnaire were analyzed through content analysis guided by the principles of completeness, representativeness, consistency, exclusivity and relevance (^{Bardin, 2009}), and by recognizing the interactive nature of the data analysis (^{Miles & Huberman, 1994}). A mixed approach and the definition of more general categories were privileged (Bardin, 2009; ^{Esteves, 2006}), articulating an inductive (emergent character of the data) (^{Cho & Lee, 2014}) and a deductive perspective through the definition of categories of analysis in line with the research goals and the theoretical framework (^{Ezzy, 2002}). The categories of analysis were semantic (Miles & Huberman, 1994).

The analysis of the participants’ discourse, combined with a critical review of the literature, supported the following central dimensions: 1) Changes in assessment practices; 2) Influence of the BP on changes in assessment practices. Each dimension originated a set of themes and categories similar to the questionnaire protocol, according to the answers “Yes”, “No” and “Maybe”: (a) structural and organizational factors; (b) factors related to the changes introduced by the legal and institutional framework; and, (c) factors related to the context of practice. In the first category, references to factors, directly or indirectly, linked to the circumstances (more or less challenging) in which the faculty’s work is developed were considered (e.g., workload, number and characteristics of students, and resources). As for the factors linked to changes introduced by the legal and institutional framework, references to legal and institutional documents were considered, as well as curricular plans, among others. Finally, regarding the context of the practice, references to the classroom and assessment practices were included.

RESULTS

The results of the EFA suggested that component 1 represented the “engagement and participation of students in the assessment”,²component 2 represented the “use of assessment by the faculty in the teaching and learning process”,³ and component 3 represented the “assessment as a process determined by external factors”⁴ (see Table A1 for detailed results). These three components reflect, on the one hand, the importance of student involvement in the assessment process (European Commission, 2013). On the other hand, they reflect how faculty use assessment, recognizing that the way they assess students (^{Brown, 2005}) and their use of assessment information (^{Earl & Katz, 2006}) have serious implications for student learning. Nevertheless, the assessment is not immune to the context, rules and procedures outside the classroom as “faculty’s decisions on assessment strategies are closely related to the opportunities and limitations offered by normative and institutional decisions” (^{Ion & Cano, 2011}, p. 167). The three components explained 46.57% of the variance. All items had high factor loadings in the respective component. The items of each factor revealed an adequate internal consistency, supporting the reliability of the obtained scores. Cronbach’s alpha was higher than 0.60 in all the three components (see Table A1).

Table 2 presents the results of descriptive statistics concerning faculty’s assessment practices. The assessment practices associated with the use of assessment in the teaching and learning process were the most used, while assessment practices determined by external factors are the least used (see Table 2). The results of the descriptive statistics also reveal a positive trend in the use of practices related to the involvement and participation of students in the assessment process.

The MANOVA indicate that there is no significant effect of the age, gender, years of experience as faculty, professional category and the frequency of peda- gogical training of the participants on the different assessment practices. Concerning the influence of study cycles on the use of different assessment practices, multivariate tests revealed the absence of statistically significant differences between the faculty who teach in the undergraduate, integrated master’s and Ph.D. programs (see Table A2 for descriptive statistics).

However, the results of MANOVA indicated the existence of statistically significant differences, with a medium effect size, between faculty who teach in master’s degree programs and those who do not (WILK’S Δ = .931, F(3, 158) = 3.876, p = .010, ηp² = .069). The results of the univariate tests allowed to conclude that the faculty who teach in this cycle of studies attached more importance to the practices associated with the involvement and participation of students in the assessment, and to the practices associated with the use of assessment by the faculty in the teaching and learning process. Less importance was attached to practices determined by external factors (see Table 3).

In regard to the field of knowledge, multivariate tests revealed the existence of statistically significant differences, although with small effect size, in the use of different assessment practices (WILK’S Δ = 0.876, F(9, 379.814) = 2.366, p = 0.013, ηp² = 0.043). The results of the univariate tests (see Table 3) and the corresponding pairwise comparisons suggested a greater use of practices associated with the involvement and participation of students in the assessment process by faculty who teach in the areas of social sciences compared to those who teach in exact sciences and engineering and technology sciences (p = 0.031). It was also possible to identify a greater use of assessment practices determined by external factors in faculty who teach in the areas of exact sciences and engineering and technology sciences compared to social sciences faculty (p = 0.029).

ASSESSMENT PRACTICES CHANGES AND THE ROLE OF THE BOLOGNA PROCESS

Assessment is a strategical asset for higher education organizations. Faculty’s right choices on assessment strategies are a key factor in students’ academic success. A good assessment can be motivating and productive for students, helping them know “how well they are doing and what else they need to do” (^{Brown, 1999}, p. 3). On the other hand, a poor assessment may be “tedious, meaningless, grueling and counterproductive” (Brown, 1999, p. 4).

Faculty were also asked about possible assessment changes after the BP was implemented. Most participants (85.9%) said that they have changed the way they assess their students over their career; 9.8% reported no change at all, and 4.3% responded “maybe” (see Table 4). When asked whether the implementation of the BP contributed to changing their assessment practices, 47.8% responded positively, 29.1% said “no”, and 23.1% responded “maybe” (see Table 4).

Source: Authors’ elaboration.

FIGURE 1 Assessment practices changes and the role of the BP 

DISCUSSION

The Bologna Process and the creation of the EHEA challenged European countries, including Portugal, to change their teaching-learning practices, focusing on student-centered pedagogies, problem-solving initiatives, and innovative assessment practices. This requires adjusting traditional teaching and learning standards based on knowledge transmission, as well as moving towards more innovative and creative forms of teaching, learning and assessment. Assessment must be differentiated to respond positively to students’ diverse backgrounds and abilities (^{Scott et al., 2014}). This study aimed to examine faculty’s assessment practices and the effects that the implementation of the BP had on those practices. A scale on the faculty’s assessment practices was used. Findings suggest that the measure presents adequate psychometric properties and might therefore be a useful tool to measure faculty’s assessment practices in HE. Findings also indicate the existence of diverse assessment practices, with a prevalence of assessment practices of the faculty, and a lower incidence of assessment practices determined by external factors. The influence of the study cycle and knowledge area on the use of different assessment practices was also identified. Indications were also found that faculty who teach in master’s programs use more practices associated with student involvement and participation in the assessment, as well as assessment practices determined more by the faculty than by external factors.

In addition, assessment practices associated with student involvement and participation in the assessment were also found to be more used by social sciences faculty than by exact sciences and engineering and technology sciences faculty. The study also identified a greater use of assessment practices determined by external factors among exact sciences and engineering and technology sciences faculty compared with social sciences faculty.

Faculty also questioned the existence of changes in their assessment practices and the possible contribution of the BP to change them. Most of the faculty reported to have changed the way they assess students over their careers; interestingly, however, less than half of participants recognize the BP’s importance in changing assessment practices. Factors related to structural and organizational aspects (e.g., number of students per class, material conditions, semesterization), to changes introduced by the legal and institutional framework (e.g., legal and institutional documents, curricular plans, among others), and to the context of practice (e.g., assessment conceptions and use) explain differences between optimistic and skeptical faculty regarding the BP’s contribution to changing assessment practices. Many participants consider that Bologna may not have been a key moment in changing - or at least challenging - their assessment practices. However, its contribution to the discussion and attention dedicated to teaching and learning and, especially, assessment issues, seems undeniable. These results highlighted the complexity of higher education scenarios (^{Zabalza, 2004}) and assessment (^{Brown & Knight, 1994}), as well as the complexity of adapting academic practices to the Bologna principles (European Commission, 2018).

The findings reinforce the importance of assessment by faculty in the teaching and learning process - for example, the degree of understanding about an issue or subject, or the role of assessment on planning future learning activities. In this context, feedback assumes particular relevance for improving students’ learning. Findings also corroborate the crucial contribution of assessment and of faculty’s right choices and strategies for the success of students’ learning (^{Brown, 1999}). The role of student engagement and participation in the assessment process is another relevant finding. From a perspective of improvement rather than measurement, the participative construction of assessment criteria contributes for an understanding of what is expected and for students’ self-regulation (^{Bonniol & Vial, 1997}). The role of curricular plans and the need to measure and attribute a classification or grade also emerges, though with less relevance, in faculty’s assessment practices.

Additionally, the participants’ voices reflect the different international perspectives in the field of assessment, sometimes within a summative and verification logic (^{Perrenoud, 1999}) and others within a student-centered assessment logic (^{Webber, 2012}), recognizing the connection between assessment conceptions and practices, as well as their influence on students’ learning (^{Gibbs, 1999}; ^{Light & Cox, 2003}).

The BP represented an opportunity of modernization and improvement of HE. However, there are two mutually opposing views: “the opinion that Bologna was an evolution towards creating better learning conditions, with curricula more adapted to reality” and the argument that Bologna represented “a simple reduction in the length of undergraduate and master’s programs that by no means altered teaching programs or methodologies” (^{Justino et al., 2017}, p. 1).

This study’s findings also point to a number of tensions regarding the impact of Bologna on assessment practices at various levels: factors related to structural and organizational characteristics, to changes introduced by the legal and institutional framework, and to the context of practice. Bologna may be seen as an opportunity to introduce some necessary changes to respond to these tensions, but the literature and the results of this research project show the need for a more systematic evaluation and a deeper reflection on the changes that have occurred on curriculum, pedagogy and assessment. The success of the BP in terms of how assessment is experienced by faculty is not evident in the data arising from this study. Nevertheless, the ambition of the BP is a positive one, and further efforts by universities are needed to mitigate a purely summative assessment approach. This research provides evidence of the role of faculty’s working conditions and workload in their assessment practices, as well as their conceptions on teaching, learning and assessment.