INTRODUCTION

In recent decades, the importance of evaluation has been paramount within national systems of basic through higher education, as well as in the realms of science, technology, and innovation. Nevertheless, it is important to recognize that the current relevance of this subject should not lead us to assume that interest in it is of recent origin. As per ^{Vught (1995)}, ^{Amaral (2009)}, and ^{Cobban (2017)}, historical records indicate that evaluation practices in higher education can be traced back to the 13^th century. According to these scholars, the first European universities employed rudimentary evaluation processes, which were based on two distinct conceptions. The first, known as the French model, conferred an external authority with the power to grant a license to teach (licentia ubique docendi) and the authority to determine the curriculum. Evaluation served as a form of accountability to the superior external authority responsible for granting approval. For example, in the early 13^th century, the bishop of Paris and the chancellor of Notre-Dame Cathedral oversaw the University of Paris. The second conception, influenced by the English model, revolved around the principle of self-governance. Universities enjoyed a considerable degree of autonomy and peers conducted evaluations. Collegial bodies possessed the right to evaluate their fellow colleagues and, if necessary, to replace them (^{Amaral, 2009}).

These divergent conceptions have given rise to distinct evaluation cultures, as noted by ^{Hicks (2012)}, ^{Lepori, Reale, and Spinello (2018)}, and ^{Ochsner, Kulczycki, and Gedutis (2018)}. Over the centuries, evaluation practices have undergone transformations, resulting in hybrid and highly heterogeneous models. It can be argued that, until the 1980s, evaluation was an internal, sporadic, and inadequately institutionalized practice. External evaluation, particularly when carried out by government entities, was perceived as a threat to scientific ethos. It was rejected based on the notion that science should be conceived and conducted according to its own assumptions, methodologies, and legitimacy criteria (^{Meulen, 1998}; ²⁰⁰⁷). Quality assessment was deemed to be the responsibility of academic institutions and peers. External evaluation was viewed as an instrument of interference in institutional autonomy, as well as a means of curbing intellectual freedom and creativity. The evaluation of the merit of scientific endeavors was conducted by peers and not by external parties (^{Molas-Gallart, 2012}).

From the 1980s, the discourse surrounding evaluation took on new dimensions. National agencies entrusted with the regulation and funding of higher education and research began to advocate for the establishment of permanent and standardized evaluation policies. According to these agencies, institutionalizing these processes would allow the assessment of the results resulting from increased public investments, while ensuring regularity, transparency, and consistency of criteria and procedures at the national level (^{Hicks, 2012}; ^{Ochsner, Kulczycki, and Gedutis, 2018}). In addition to being an internal and exclusive concern of universities and researchers, evaluation gradually assumed a prominent role within national systems of higher education, science and technology (^{Amaral, 2009}; ^{Verhine and Freitas, 2012}). Through their regulatory and funding bodies, nation states progressively undertook the responsibility of conducting periodic evaluations, which not only influenced research and higher education policies but also had an impact on resource allocation. This process led to the so-called “performance-based funding system” (^{Ochsner, Kulczycki, and Gedutis, 2018}, p. 1235). The nexus between performance and funding caused profound changes in the management of universities and research institutions (^{Castro, 1999}; ^{Sobrinho, 2000}; ²⁰⁰³; ^{Amaral, 2009}; ^{Afonso, 2013}).

Over the past few decades, numerous countries have established institutionalized evaluation systems. While some have chosen centralized and standardized models based on metrics, others have implemented decentralized models that are less quantitative, emphasizing the idea that evaluation should be conducted by the institutions themselves to reinforce their institutional missions and guide strategic planning.

In this context, the focus of the present study is on the evaluation system in the Netherlands. The Netherlands was among the first European countries to adopt a formal system for assessing the quality of research and higher education.¹ The Dutch experience has served as a benchmark for other nations and institutions, including Brazil. Their decentralized, bottom-up model is structured around the principles of autonomy, self-evaluation (SE), and institutional planning (^{Weert and Boezerooy, 2007}; ^{Drooge et al., 2013}). The recognition this model has gained within the academic and scientific community, as well as its notable position in internationally indexed scientific production rankings (14^th place according to the Web of Science, Clarivate, n. d.), highlights the significance and contributions of evaluation policies in consolidating quality.

Specifically, this article aims to:

The examination of the key characteristics and specificities of the Dutch model is complemented by a case study. This case study pertains to a recent SE initiative conducted between 2021 and 2022 by the Center for Science and Technology Studies (CWTS) at Leiden University. Drawing from this study, we highlight certain aspects that can provide valuable insights for enhancing the higher education evaluation system.

REGULATORY FRAMEWORKS FROM A HISTORICAL PERSPECTIVE

According to the research conducted, the Netherlands initiated its first efforts to establish norms and guidelines for evaluating the quality of research in the late 1960s. In 1969, the Advisory Council for Science Policy recommended the formation of expert committees to evaluate research programs funded by the government. In the early 1970s, committees were appointed for areas such as education, social policy, and environmental planning, followed by chemistry, biochemistry, biology, and physics in 1976. These committees comprised representatives from three stakeholder groups: researchers, government members, and stakeholders. However, lack of clarity regarding objectives, procedures, and evaluation criteria hindered the success of these initial endeavors (^{Meulen, 1998}; ²⁰⁰⁷). In 1979, the Ministry of Education and Sciences introduced new guidelines for organizing research in institutions. Expert committees were tasked with evaluating research programs, which became the focal point of assessment (^{Goedegebuure and Westerheijden, 1991}).

Evaluation experiences throughout the 1970s provided valuable insights for the development of the first regulatory framework for evaluation in the Netherlands. The Conditional Financing (CF), published in 1982, laid down general guidelines for establishing the national evaluation system. According to ^{Goedegebuure and Westerheijden (1991)}, the CF can be seen as the initial attempt to introduce a formal system for assessing the quality of higher education and research. Its objectives were to promote quality, improve transparency, and evaluate the results of government and other funding agency investments. The document solidified research units as central organizing cores within institutions (^{Rip and Meulen, 1995}; ¹⁹⁹⁶; ^{Meulen, 1998}; ²⁰⁰⁷). Additionally, the CF:

The CF introduced a mandatory external evaluation, conducted through the peer review system. As described in the document, the researchers were assigned the responsibility of evaluating academic quality and research relevance due to their extensive and specialized training. The presence of external and independent evaluators was considered crucial to ensure legitimacy, improve transparency, and provide reliable information to the academic community and the institutions involved (^{Weert and Boezerooy, 2007}; ^{Drooge et al., 2013}).

The appointment of external committees was the responsibility of the Royal Netherlands Academy of Arts and Sciences (KNAW), a national scientific association. The peer review system served to protect the evaluation process from external interference, particularly from funding agencies. According to the document, evaluation should not be conducted by funding agencies or the institutions being evaluated (^{Rip and Meulen, 1995}; ¹⁹⁹⁶; ^{Meulen, 1998}).

Furthermore, the CF brought about changes in funding policies. The sole criterion of enrollment numbers was no longer used to determine institutional funding (^{Goedegebuure and Westerheijden, 1991}). Evaluation results in terms of research quality and societal relevance began to influence institutional budget allocation (^{Meulen, 1998}; ²⁰⁰⁷; ^{Weert and Boezerooy, 2007}).

During the 1980s, the Ministry of Education, Culture and Science published several supplementary documents aimed at refining and enhancing the policies established by the CF. The principal document, titled Hoger Onderwijs: Autonomie en Kwaliteit (^{OCW, 1985})²—commonly known as the Hoak Document—reinforced institutional autonomy and introduced a retrospective quality assessment system. Instead of imposing interventionist policies, the document encouraged the development of institutional evaluation policies and the strengthening of external evaluation processes. Universities were directed to implement more efficient and professional institutional management practices (^{Goedegebuure and Westerheijden, 1991}). Alongside the promotion of autonomy, the Hoak Document established new guidelines for funding and institutional planning. Institutions were challenged to operate in a more professional manner and align with societal needs. In exchange for increased autonomy, universities were required to establish institutional evaluation policies and ensure transparency of results (^{Goedegebuure and Westerheijden, 1991}; ^{Goedegebuure et al., 1994}; ^{Kaiser and Weert, 2006}; ^{Drooge et al., 2013}).

In 1992, the government enacted a new law to regulate the relationships between the government, higher education institutions, and research institutes. The Hoger Onderwijs en Wetenschappelijk Onderzoek law (^{OCW, 1992})³ further solidified autonomy and the role of institutions in the evaluation process. The law reduced state control and limited government interference in the operational aspects of evaluation. State intervention was restricted to specific circumstances. According to the law, the government should only intervene in cases of identified deficiencies, misallocation of resources, or discrepancies between objectives and results. Rather than conducting the evaluation itself, the government was responsible for ensuring the macro-efficiency of the system.

According to ^{Weert and Boezerooy (2007}, p. 14), the law established four general principles to guide government action:

The document emphasizes that evaluation is a component of institutional planning. Institutional policies should incorporate the participation of external and independent evaluators, preferably affiliated with other institutions (^{OCW, 1992}, sec. 1.18). Evaluation is not an end in itself, and should not be used as a tool for external control. Instead, it should be seen as a means to enable the higher education system to respond more effectively and decisively to the needs and changes of society (^{Weert and Boezerooy, 2007}, p. 13). The document provides the legal framework for institutions to develop their own evaluation policies.

In 1993, the Association of Dutch Universities (Vereniging van Universiteiten—VSNU),⁴ in collaboration with the Netherlands Organization for Scientific Research (Nederlandse Organisatie voor Wetenschappelijk Onderzoek—NWO)⁵ and the Royal Netherlands Academy of Arts and Sciences (Koninklijke Nederlandse Akademie van Wetenschappen—KNAW),⁶ formulated the initial general protocol for evaluating research quality. After testing and refinements, the protocol was adopted by all universities and research institutions in the Netherlands. The coordination of the evaluation process was entrusted to the VSNU.

In 1998, the second evaluation protocol, covering the period 1998–2003, was approved (^{VSNU, 1998}). The following year, in 1999, the three main regulatory and funding agencies for higher education and research in the Netherlands (VSNU, NWO and KNAW) established a working group tasked with reviewing and improving evaluation policies. The final report, completed in 2000, served as a basis for the development of the third protocol. Subsequently, four additional regulatory frameworks were developed between 2003 and 2021, corresponding to the periods 2003–2009, 2009–2015, 2015–2021 and 2021–2027. From 2003 onward, the protocols were renamed the Standard Evaluation Protocol (SEP). In 2021, the term “standard” was replaced by “strategy” to reinforce a key principle of the Dutch system. The essence of the evaluation lies in the objectives and goals that the unit (institution) has set for itself. The evaluation is based on what the unit aspires to achieve. Therefore, the evaluation process is considered a form of SE. The strategy precedes the evaluation. The mission and objectives should be clearly formulated, known, shared, and consistently pursued (^{VSNU, 1993}; ¹⁹⁹⁸; ^{KNAW, 2008}; ^{Drooge, 2021d}).

The series of regulatory frameworks discussed above have established the foundation of what is commonly referred to as the Dutch model of evaluation. These frameworks have defined the principles and guidelines for the institutionalization of periodic evaluation policies and processes at the national level. While each protocol reflects the context and requirements of its respective period, there are clear threads of continuity that run through them. The fundamental characteristics of the current evaluation model were already present in the SEP 2003–2009 (^{VSNU, KNAW, and NWO, 2003}). The subsequent protocols, covering the periods 2009–2015, 2015–2021, and 2021–2027, have primarily introduced incremental changes, further solidifying the system. As noted by ^{Drooge et al. (2013)}, these successive protocols have contributed to the institutionalization of a robust and lasting quality evaluation system.

Examining the evolution of the protocols over the past four decades allows for the identification of the key principles and guidelines that have shaped the policies and processes of research and higher education evaluation in the Netherlands. The presence of these lines of continuity enables us to analyze the similarities between the protocols and to understand the changes and improvements implemented throughout the years.

THE INSTITUTIONAL DESIGN OF THE EVALUATION SYSTEM

The ongoing changes and continuous improvements in the Dutch evaluation system have solidified a bottom-up model of evaluation. Since the publication of the Hoak Document (^{OCW, 1992}), all subsequent regulatory frameworks have reinforced the principle that evaluation should be conducted by the institutions themselves, which engage in research and offer graduate programs. Evaluation is considered an inherent responsibility of institutional autonomy and is closely related to institutional planning and quality improvement policies. Quality evaluation cannot be delegated to funding agencies or external institutions. Therefore, in the Netherlands, there is no national agency, public or private, tasked with conducting quality evaluation processes for research and graduate education. Evaluation is mandatory every six years, but institutions and units are not required to undergo evaluation in the same year, and there is no unified national schedule.

The system is decentralized, giving institutions and units the autonomy and competence to establish their own internal evaluation policies and procedures. However, this decentralization and exercise of autonomy occur within a framework of nationally established guidelines provided by protocols published every six years. These protocols, known as the Strategy Evaluation Protocol (SEP), serve as the primary regulatory frameworks for evaluation.⁷ According to the current SEP (^{VSNU, KNAW, and NWO, 2020}), the main objective of the protocols is to improve the quality and societal relevance of research, while fostering a continuous dialogue about the actions needed to improve quality and increase transparency and accountability to society, funding agencies, and the government. In addition to setting general guidelines, the protocols provide detailed procedures to guide the evaluation process. They are intended for all stakeholders involved in the evaluation, including researchers, unit directors, university managers, council members, science and technology managers, evaluation committee members, committee secretaries, and graduate students, among others.

As mentioned earlier, a significant change was introduced in the legal framework in 1992 to minimize external interference and control. With the publication of the document Hoger Onderwijs en Wetenschappelijk Onderzoek [Higher Education and Scientific Research] (^{OCW, 1992}), the Ministry of Education, Culture, and Science ceased to issue guidelines and norms regarding the evaluation process. The regulatory authority was transferred to the associations representing the institutions involved in the evaluated activities. The protocols began to be jointly developed by the three major scientific and academic associations in the Netherlands: VSNU, NWO, and KNAW (^{Drooge et al., 2013}). The changes implemented over the past four decades have solidified the principle that evaluation should be conducted by peers, based on the assumptions and criteria that guide academic ethos. Funding institutions should not be responsible for the evaluation. Therefore, the separation between funding and evaluation is another crucial characteristic of the Dutch model.

The SEP undergoes a revision every six years. These periodic revisions aim to adapt the protocols to the needs of several types of higher education institutions, research institutions, and units. Consequently, the SEP is not the only evaluation protocol in effect in the country. There are specific protocols for research universities, universities of applied sciences, and academic and nonacademic research institutes (^{Drooge et al., 2013}). It is important to note that the Dutch higher education system encompasses fourteen research universities (Universiteiten-WO), four small, specialized universities, numerous universities of applied sciences (Hogescholen-HBO), and an open university (Open Universiteit) (^{Drooge, 2021a}; ^{VSNU, 2022}).

An important characteristic of the evaluation protocols in the Netherlands is the absence of a distinct separation between research and graduate education. Doctoral programs are considered integral components of research programs and are evaluated as such. Therefore, the Strategy Evaluation Protocol provides guidelines for assessing the quality of research without the need for separate regulatory frameworks for evaluating graduate education.

Furthermore, it is worth noting that master's degree programs are approached differently in the current Dutch model compared to Brazil. In the Netherlands, as well as in many European countries, master's programs are part of the sequential educational process following undergraduate studies. This approach is in line with the Bologna Process and the adoption of the European Credit Transfer System, which promotes harmonization and standardization of higher education across Europe.

PRINCIPLES AND OBJECTIVES OF EVALUATION

The organization of the Dutch evaluation system, as described above, is founded on principles and overarching objectives that have guided evaluation policies since the publication of Hoger Onderwijs en Wetenschappelijk Onderzoek (^{OCW, 1992}) and the development of the first protocol in 1994. The regulatory frameworks developed over the years have defined and solidified the principles shaping the system's organization and functioning. Evaluation has been conceptualized based on key assumptions, including university autonomy, external evaluation through the peer review system, SE, accountability, quality improvement, and separation of funding and evaluation.

Based on these principles, the protocols have established a clear distinction from performative evaluation conceptions that view evaluation as a means of performance evaluation, outcome control, accountability, ranking generation, and the development of indicators for the allocation of rewards and incentives, such as scholarships and financial resources.

The mentioned principles are summarized in Chart 1, serving as the foundation for the objectives set forth by the protocols over the decades. These objectives reflect the system's commitment to improving quality, ensuring accountability, and enhancing the societal relevance of research (^{Drooge, Jong, and Smit, 2022}).

Chart 1 highlights not only the similarities and continuity in the objectives but also the qualitative changes that have occurred over time. The SEP 2021–2027 places a stronger emphasis on SE. In this context, the core of the evaluation process is based on the objectives, goals, and strategies established by the units⁸ themselves over the six-year period. Evaluation is not an externally imposed exercise, but rather a contextual and formative process. As emphasized by ^{Drooge (2021a}; ^b; ^c; ^d), it is an integral part of a continuous cycle of quality assurance and improvement. The evaluation process fosters an ongoing dialogue between the evaluated unit and various governing bodies, particularly regarding strengths, weaknesses, and areas for improvement that need to be addressed.

DIMENSIONS AND CRITERIA OF EVALUATION

The evaluation criteria, along with the principles and objectives discussed above, reflect the evolution and transformations of higher education, science, and technology policies in the Netherlands. These changes are a response to the evolving demands of society, the scientific community, and institutions. Therefore, the evaluation criteria express what society expects from science, researchers and their academic and scientific institutions (^{Spaapen and Drooge, 2011}).

Chart 2 highlights the incremental changes introduced over time. A significant change was the reduction in the number of evaluation criteria from four to three. In 2015, the criterion of “productivity” was removed. This change was based on a proposition that had been advocated by the scientific community for several years (^{Drooge, Jong, and Smit, 2022}). Additionally, the criterion of “relevance” underwent improvements over the decades. Starting from the SEP 2009–2015, it was expanded and became known as “societal relevance”. This change broadened the categories that guide the analysis of research impacts on society, encompassing not only scientific and socioeconomic dimensions but also cultural, social, and political impacts (^{VSNU, KNAW, and NWO, 2009}). In 2015, the SEP 2015–2021 introduced further elements. It required that evaluation committees assess the quality and relevance of research based on the specific areas of operation and target audiences of each unit. Evaluation should consider the contributions of research to society, particularly in areas such as economic development, innovation, education, culture, health, sustainability, and more (^{VSNU, KNAW, and NWO, 2009}; ^{Drooge et al., 2013}).

The evaluation criteria used in the Dutch system cover both quantitative and qualitative aspects, allowing for a comprehensive assessment of research and higher education. It is important to note that these criteria should not be applied rigidly. Institutions, councils, and committees have the autonomy to adapt and adjust according to the characteristics and objectives of each unit being evaluated. According to the SEP 2021–2027 (^{VSNU, KNAW, and NWO, 2020}), the criteria are flexible tools; they should be interpreted according to the goals, strategies, and SE policies of each unit.

GUIDING PRINCIPLES OF SELF-EVALUATION

As previously highlighted, SE is a fundamental aspect of the Dutch evaluation system and distinguishes it internationally. It permeates and articulates the principles, objectives, criteria, and procedures of the evaluation process. It serves a dual role as both a principle and a method, defining the objectives and guiding the organization and procedures of the evaluation.

Based on the principles of autonomy, institutional planning, and accountability, the Dutch evaluation system has institutionalized evaluation as a practice of self-management conducted from within and from the bottom up. The units being evaluated are simultaneously “subjects” and “objects” of the process. External evaluation does not overshadow internal evaluation. As emphasized in the SEP 2021–2027 (^{VSNU, KNAW, NWO, and 2020}), the core of the evaluation lies in the objectives, goals, and strategies set by the unit itself to maintain and improve quality, societal relevance, and viability. Goals and strategies are central elements of SE, allowing the unit to analyze the results achieved based on the previously established objectives and assess performance considering the mission and purposes it has established for itself. According to the SEP 2021–2027, SE enables reflection “[…] on the strategies adopted by the unit, as well as the effects they have produced” (^{VSNU, KNAW, and NWO, 2020}, p. 19).

SE is, in this sense, a process related to the planning and management of the unit. As emphasized by ^{Drooge (2021a}; ^b), it is contextual and formative. In addition to considering the stages of development of the units, SE allows for the identification of progress, potential, weaknesses, and threats. In the words of ^{Drooge (2021b)}, SE “[…] is not focused on research itself, but on the unit's strategy regarding research.” For this reason, the SEP 2021–2027 (^{VSNU, KNAW, and NWO, 2020}, p. 19) states that SE begins when the unit “[…] explicitly articulates its strategy and objectives.”

As highlighted earlier, the unit is both the subject and the object; it is both the starting point and the end point of the evaluation process. The results of the evaluation are returned to the unit in the form of a report to guide the planning and strategic actions of the next cycle. SE is, therefore, both an end and a means. Instead of being a specific, isolated, and independent stage, it is the structuring dimension around which the distinct stages are articulated and fed into one another.

SELF-EVALUATION AS A PROCESS: THE CENTER FOR SCIENCE AND TECHNOLOGY STUDIES EXPERIENCE

The experience described here provides valuable insights into the Dutch model of research and graduate evaluation, as implemented by the Center for Science and Technology Studies (CWTS) at Leiden University.

This experience is particularly relevant, as the CWTS is a renowned research center in the fields of research evaluation and scientometrics. As a result, SE provided opportunities for CWTS researchers to explore new methods, techniques, and differentiated approaches. These efforts to construct a robust evaluation process, based on the combination of quantitative and qualitative methods, were reported by the working group that supported the center's management in producing the report and related activities (^{CWTS, 2022}), which also formed part of the final evaluation report completed in 2022 (ibidem).

The main stages of the SE process conducted by the CWTS between 2021 and 2022 can be summarized as follows, based on the reports and analysis of procedures outlined in the Strategy Evaluation Protocol 2021–2027 (^{VSNU, KNAW, and NWO, 2020}).

CONCLUSION

The Netherlands was one of the first European countries to implement a formal system for evaluating the quality of research and graduate education. The policies and changes implemented over the past four decades have established a sui generis, enduring, and stable system. Like all national evaluation systems, the Dutch model was conceived and improved on the basis of the country's characteristics and needs. It is part of a broad and complex political, cultural, social, and academic process. In addition to reflecting evaluation concepts, it also expresses the guidelines that the country has established for organizing higher education and the national science and technology system. Unlike other nations that have implemented centralized, standardized, performance-based evaluation systems, the Netherlands has consolidated a decentralized and heterogeneous model based on the principle that evaluation should be conducted based on the mission, goals, and strategies that the units being evaluated establish for themselves. Evaluation is therefore an internal, formative, contextual and participatory practice.

The Dutch system has been a reference and inspiration for numerous institutions and countries, including Brazil. Researchers and evaluation agencies have studied the model to understand its specificities and identify contributions to improving national evaluation systems. As this purpose also motivated the present study, we consider it important to conclude this article by highlighting some aspects that can serve as subsidies for improving the postgraduate evaluation system in Brazil.

The first aspect concerns the protagonism exercised by the institutions and the units being evaluated. Evaluation is an ongoing institutionalized practice of self-awareness and self-management. It is related to the exercise of institutional autonomy and strategic planning. It is not an instrument of state regulation, external control, or the production of rankings, performance indicators, and comparisons. There is no national institution responsible for organizing, coordinating, and implementing the process at a national level. National evaluation protocols invariably attribute the competence and responsibility for evaluation to the institutions themselves. This trust has been reaffirmed and strengthened over the decades.

Second, the Dutch system is based on the principle that evaluation should prioritize the mission, goals, and strategies established by the unit being evaluated. The evaluation objectives should not be disconnected from strategic planning. Instead of assessing whether the unit has fully or partially achieved nationally established performance standards, the evaluation process seeks to analyze the results considering the goals and strategies previously established. Units are both the starting point and the destination. Through self-evaluation, they identify their strengths, weaknesses, and areas for improvement that need to be implemented.

Third, the evaluation aims to respect and strengthen the identity of each unit through a multidimensional approach. Units have the freedom to define the dimensions they wish to consider in their own evaluation, adopting criteria and indicators tailored to support each individual narrative, without generating standardized results for the entire system. Instead of promoting homogenization, the process preserves diversity and differentiation, even allowing for optional comparisons between units.

The fourth important aspect concerns the periodicity of the evaluation. Since 2003, the cycles have been set at six years. Longer periods facilitate the development and monitoring of units’ strategic planning.

Indeed, while not perfect, the Dutch model seeks to integrate complex and challenging dimensions. Through SE, the system aims to promote balance and complementarity between qualitative and quantitative dimensions, internal and external evaluation processes, formative and summative dimensions, institutional autonomy and accountability, and processes of standardization and differentiation.