SciELO - Scientific Electronic Library Online

 
vol.57 número53Educación para la comprensión humana: desarrollo de la intersubjetividad desde la complejidadFormación docente y prácticas pedagógicas multiculturales críticas índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Compartir


Revista Educação em Questão

versión impresa ISSN 0102-7735versión On-line ISSN 1981-1802

Rev. Educ. Questão vol.57 no.53 Natal jul./sept 2019  Epub 19-Sep-2019

https://doi.org/10.21680/1981-1802.2019v57n53id17097 

Article

Higher education quality policies in Germany and United Kingdom: quo vadis?

Cleber Augusto Pereira3 
http://orcid.org/0000-0002-7704-2343

Joaquim Filipe Ferraz Esteves de Araújo3 
http://orcid.org/0000-0001-8531-6036

Maria de Lourdes Machado-Taylor4 
http://orcid.org/0000-0003-4774-2397

3Universidade do Minho (Portugal)

4Centro de Investigação em Políticas do Ensino Superior (Portugal)


Abstract

The objective of this study is to discuss the accreditation systems in Higher Education (HE) currently in Europe. The specific goals include describing the context of HE in Germany and the UK, identifying the main policies implemented. The studies are systematically mapped and contextualized and they will analyse quality assurance and evaluation. This research describes the assessment of the social phenomenon and the accreditation of HE based on the performance of European rating agencies. This paper will also present the main points of action of the quality policy and the quality of HE agencies and their development. It was created a concept mapping of HE in these countries and it was also discussed the characteristics, restrictions and evolution of the processes of accreditation, evaluation and peer review. The main points of action of quality policies and quality agencies in HE, and their current situation of development were demonstrated.

Keywords: Higher Education; Quality; Accreditation; Assessment

Resumo

O objetivo deste estudo é discutir os sistemas de acreditação na Educação Superior (ES) atualmente na Europa. Os objetivos específicos incluem a descrição do contexto da ES na Alemanha e no Reino Unido identificando as principais políticas implementadas. Os estudos foram sistematicamente mapeados e contextualizados e analisam a garantia e a avaliação da qualidade. Esta pesquisa descreve a avaliação do fenômeno social e a acreditação da ES com base no desempenho das agências de avaliação da ES europeia. Este artigo também apresentará os principais pontos de ação das políticas de qualidade, as iniciativas das agências de qualidade do ES e seu desenvolvimento. Foi criado um mapeamento conceitual da ES nesses países e também foram discutidas as características, restrições e evolução dos processos de acreditação, avaliação e revisão por pares. Os principais pontos de ação das políticas de qualidade e agências de qualidade em ES e sua situação atual de desenvolvimento foram demonstrados.

Palavras-chave: Ensino Superior; Qualidade; Acreditação; Avaliação

Resumen

El objetivo de este estudio es discutir los sistemas de acreditación en la Educación Superior (ES) actualmente en Europa. Los objetivos específicos incluyen la descripción del contexto de la ES en Alemania y el Reino Unido identificando las principales políticas implementadas. Los estudios fueron sistemáticamente mapeados y contextualizados, y analizan la garantía y la evaluación de la calidad. Esta investigación describe la evaluación del fenómeno social y la acreditación de la ES en base al desempeño de las agencias de evaluación de la ES europeas. Este artículo también presentará los principales puntos de acción de las políticas de calidad y la calidad de las agencias de ES y su desarrollo. Se creó un mapeo conceptual de la ES en esos países y también se discutieron las características, restricciones y evolución de los procesos de acreditación, evaluación y revisión por pares. Los principales puntos de acción de las políticas de calidad y agencias de calidad en ES y su situación actual de desarrollo han sido demostrados.

Palabras clave: Enseñanza Superior; Calidad; Acreditación; Evaluación

Introduction

Until the 1990s, both the European and the South American Higher Education Institutions (HEI) operated in a context of minimal competitiveness, in which the demand was greater than the offer. With regards to the European Public HEI, the process of management reform (MONTEIRO; PEREIRA; SOUZA, 2015) was directed towards a lack of resources, which had an impact on demand. This new reality of higher education rooted on neoliberal precepts led HEI to move from a concern previously centred exclusively on teaching to a new one focused on providing services that were competitive and dynamic, and in which were involved academic, administrative and methodological aspects.

In order to adapt this new reality, HEI were pressured to incorporate and seek new quality and excellence patterns. At the same time, they had to respond to the market in an increasingly faster and more efficient way in order to fulfil the expectations of their public and other interested parties, such as stakeholders, community, professors and students.

In their constant search for excellence, quality becomes the differential factor in HEI and their survival in the market. There is a range of requirements that must be met in order to attain excellence and a public profile: from standards established by governmental agencies that provide monitoring, accreditation and auditing to other differentiating factors, including course management, infrastructures and scientific production of professors.

The overall objective of this study is to discuss the accreditation systems in HE currently in force in Europe. The specific goals include describing the context of HE in Germany and the UK; identifying the main policies implemented in HE, with a special emphasis in quality; creating a concept mapping of the situation in these countries.

This study conducts a descriptive survey, in which we seek the what is provided by Knupfer and Mclellan (1996, p.1197) portraying "[...] what exists today and now in relation to a problem or phenomenon" (MERTENS, 1998, p. 174). A correlational analysis was carried out that aims to find and assess the relationship between the variables, represented here by the agencies and the policies in place in HE (ANDERSON, 1999; MELTZOFF, 1998).

Quality assurance and evaluation of higher education

This section discusses the need for more refined techniques for quality assurance and the due need for publication of these results in the media. The styles used in Europe and the United States will also be studied. The following are an overview of Germany and the United Kingdom.

Studies by Massy (1997; 2010) reiterate the need for more refined techniques for quality assurance in HE and they are classified in three categories: accreditation, standard used in the USA; evaluation, according to the European style; and process of peer review, which can be denominated audit of the quality of the process, which generally employs a combination of indicators of performance, self-study and peer review.

Accreditation, Evaluation and Quality Audit

Accreditation has been an important element in HEI in the United States (HARVEY, 2002; BRITTINGHAM, 2009; HARVEY, WILLIAMS, 2010) and within the last years has taken on growing significance around the world. In the United States the term accreditation refers to the results of a public process which has as a goal determining whether HEI follow the quality standards established. Some researchers complement the meaning of this term by saying that improvements in the methodology of teaching may reflect positive outcomes in accreditation (PRINGLE, MICHEL, 2007; KELLEY, TONG, CHOI, 2010).

According to Massy (1997), the overall characteristics of the adopted styles of accreditation in the USA operate exclusively in education. Institutional accreditation is more common on undergraduate courses, comparing the performance of HEI in relation to predetermined standards established by governmental agencies or accrediting agencies. The process assesses if the objectives of an institution are appropriate for the degree or level in question. Given that it is a certification function, it should be undertaken by an outside agency.

The process of accreditation is carried out in cycles and its objective is to ensure that minimal standards of quality are being followed by HEI. The outcome of the accreditation process has to be published, which is a necessary step for the implementation of the certification function (HEQC, 1994).

Institutional accreditation agencies in the United States, where the task is performed by non-governmental independent agencies (private institutions), according to Massy (1997) traditionally objected to full disclosure. However, as deliberated by the Higher Education Accreditation Council (HEAC), as the body responsible for accreditation in the USA, disclosure is mandatory because confidentiality impairs public responsibility (EATON, 2012).

The evaluation usually focuses on the quality of specific educational activities. The overall evaluation uses some accreditation parameters, combining performance indicators, self-assessment and peer review. This balanced combination of performance indicators and HEI being visited by evaluators is desired, however, due to the great amount of HEI in existence, it is difficult to be fully attended.

Firstly, the evaluation takes place in a specific course or programme and afterwards the entire HEI is evaluated as a whole. As defined by the Higher Education Funding Council for England (HEFCE, 1992) in the United Kingdom the evaluations happen separately for teaching and research. Most recently, it was disclosed that the main objective is the production of quality profiles through clarity, coherence, continuity, credibility, efficiency, neutrality, parity and transparency (HEFCE, 2008).

The academic, managerial and pedagogic dimensions with emphasis on the market established by Brennan and Shah (2000) are similar to the dimensions used by the accreditation agencies: the Faculty, Coordinator, Educational Project and Courses infrastructure.

The evaluation process may be organized by the country's government, an institutional consortium, or the HEI. Massy (1997) mentions the example of Great-Britain and Hong Kong, where the quality of teaching is defined by the mission of the HEI and not if it follows some "gold standard" of academic excellence. In his most recent analysis, Massy (2010, p. 203) criticizes once more the way the evaluation is held in the country, carried out by Hong Kong's Grants Committee (HKGC) which aims to occur without taking away power from institutions, violating their autonomy or spending much in relation to the results achieved.

For institutional accountability to occur it is necessary that the outcomes of the evaluation are published and expressed in such a way that allows comparison between institutions. Silva-Trivino and Ramirez-Gatica (2004) reported on the Central American University Higher Council (CSUCA) has addressed evaluation processes orientated to external accountability and to an eventual regional accreditation system. Experiences are analyzed with different focuses: institutional and programme.

An important fact presented by Harvey (2008), mentions that in most cases in which quality assurance is focused on improvement, political pressure mechanisms can occur in the sense of "forcing" an "accountability" orientation into the process. Per Amaral and Rosa (2008) these types of definition problems can create substantial transparency and comparability problems in the evaluation.

Regarding the restrictions in following the quality assurance, the ENQA (2006) has a clear point of view when it states:

Where an agency is found to be either partially compliant or non-compliant with a criterion, the reason for this should be explained. Full or substantial compliance may be impossible for some agencies, owing to restrictions placed on them by the very nature of their work and/or legislation in place in their country (ies) of operation. When considering such cases, the ENQA Board will take mitigating circumstances such as these into account (ENQA, 2006, p. 6).

External evaluation occurs in time cycles of five to ten years, slightly shorter cycles than those of accreditation. Long periods of evaluation of the cycle are attributed to the costs of visitation and the large number of units of evaluation (MASSY, 1997). In some European countries in which the Bologna Process is in place, the external evaluation process has only demanded the presentation of self-assessment reports for the internal quality assurance dimension (RAUHVARGERS; DEANE; PAWELS, 2009).

Silva-Trivino and Ramirez-Gatica (2004) reported on the Central American University Higher Council (CSUCA) has addressed evaluation processes orientated to external accountability and to an eventual regional accreditation system.

In the quality audit, the review of the processes of quality is centered in the analysis of internal quality and the implementation of systems aiming at improving processes. Studies from different parts of the world report that inducing training may lead to an improvement in teaching and in learning in HEI (DILL, 2000, 2000a; MASSY, 2005, 2010).

These procedures do not evaluate the quality in itself; instead they focus on the processes that are accredited to produce quality and the methods that are used by HEI to achieve quality. As Massy (1997) sees it, the quality of the process is based in the principle that good professionals, working with adequate resources, and using good processes will produce good results, but defective processes may prevent even good professionals with abundant resources from producing optimal results.

The accreditation and evaluation agencies in Germany and the UK have participated in several meetings of the Quality Audit Network (KASTELLIZ; MITTERAUER, 2014), which has established a regular dialogue on models of quality audit. Currently the group is comprised of 12 European quality assurance agencies and has published a key study named "[…] an evaluation of the strengths and weaknesses of the quality mechanisms established by an institution itself to continuously monitor and improve the activities and services of either a subject, a programme, the whole institution or a theme" (COSTES; CROZIER; CULLEN; GRIFOLl, 2008).

Meade and Woodhouse (2000) reported the review of the New Zealand Academic Audit Unit by an independent group. The review panel concluded that the quality assurance procedures, including self-review, have been a major stimulus for introduction of effective quality systems. Trust and mutual respect have been established and there are promising signs that universities are beyond compliance. According to Kanji (1998) in order to achieve excellence, many HE organizations have adopted the scorecard approach to excellence in business to improve the processes. Kanji (1998) states that the value of the interested party is the outcome of organizational excellence, which can only be achieved by accomplishing the key elements i.e. the excellence of processes, organizational learning and interest of the interested party.

Insuring the quality of the process may take place inside HEI and, by becoming continuous, it tends to improve institutional quality; since, in theory, both the actions and the results produced are assessed, therefore allowing the institutionalization of the actions to be implemented to guarantee quality.

The European context

The studies presented in this section contextualize the quality and evaluation scenarios in HE in Europe. To that end, it was carried out a survey of the quality scenario involving Germany and the United Kingdom.

Germany

Germany has excelled as one of the top five most competitive advanced economies, being placed fourth on the Global Competitiveness Index 2015-2016 (SCHWAB; SALA-I-MARTIN; SAMANS; BLANKE, 2015). The country's innovation system is characterized by high levels of company spending on R&D, a supportive research environment, including business collaboration with universities, and strong scientific research institutions.

Germany's 2015's evaluation by the Global Competitiveness Index, when relating to the Higher Education and Training (5th evaluated pillar), has allowed it to reach the 17th position in the 140 countries evaluated by the World Economic Forum (SCHWAB; SALA-I-MARTIN; SAMANS; BLANKE, 2015).

In Germany were identified specific studies on tertiary education systems' in HEIs. We selected studies that addressed the evolution of this issue in Germany in the period between 1997 and 2014 (EL HAGE, 1997; BERNER, RICHTER, 2001; HÜFNER, 2003; WARNING, 2004; FANDEL, 2007; HARRIS HUEMMERT, 2008; KEMPKES, POHL, 2010; ORR, HOVDHAUGEN, 2014).

According to El Hage (1997), an analysis on the situation of evaluation in HE brings forward a discouraging reality in Germany, which was exposed by reports and researches carried out in the 80s and 90s. These showed that if we compared a German student with other European students, we would notice that they needed 3 to 5 extra semesters to graduate (EL HAGE, 1997). This led to a decrease in public investment in scientific and educational funds on the part of the national government. There were also serious problems when it came to comparing performance between universities, which began to follow the recommendations of standards suggested in the Conference of Rectors and Vice Chancellors of the Universities and HEI, in the annual reports on education (EL HAGE, 1997). The foundation of the Centre for Development of Higher Education (Zentrum) in Giitersloh was an initiative to promote quality in HE in German universities through internal and external evaluations of faculty performance. Although some tools were developed to evaluate courses, their implementation, significance and publication of comparable results were still very controversial (EL HAGE, 1997).

Berner and Richter (2001) reported that the 16 Germans states had agreed to the establishment, the bachelor's and master's degree programmes, to be validated via accreditation agencies approved by a national accreditation council. This implementation of accreditation procedures marks a fundamental shift in the relationship between HEI and the state and collaborate to offer further possibilities for the development and modernization of the German HE.

Hufner (2003) explained the complex process of decision-making in HE when it comes to legal and administrative requirements, planning and financial issues in the Federal Republic of Germany. He analyzed the increased privatization of HE and the legal and financial problems that followed. The introduction of new working systems based in performance indicators boded great changes in the status and in the legal management structures of HE. Although these changes led to a drastic increase in the number of students between the years 1975 and 2000, reaching approximately 1.8 million (HUFNER, 2003), the vacancies for students did not sustain the growth and neither did the numbers of professors.

Some public statements defined this complex moment in German HE and are presented here. Michael Daxner, in 1996, former vice-chancellor at the University of Oldenburg, stated that "the university is an extremely complicated system, in which some parts work extremely well, some not at all, and others remain stagnant"; Dieter Simon, vice-chancellor of Wissenschaftsrat (Science Council), said that "[…] universities are rotten by his side" (HUFNER, 2003, p. 148).

The 1998 amendment to Hochschulrahmengesetz (Federal Basic Law of Higher Education) reforms the HE system in Germany, creating competition and differentiation through deregulation, performance guidelines, and the introduction of performance incentives. These reforms are part of the policy to increase competitiveness in HE in Germany and provide important incentive measures, such as public funding of HEI, resource allocation, at institutional and departmental level; in addition, they became directly based on performance indicators (HUFNER, 2003).

According to Santos (2001, p. 61-62) the Standing Conference of the Ministers of Education and Cultural Affairs (KMK) promotes a certain alignment of policies between the different States, although its deliberations are not binding and will only have an effect on the HEI after being reflected in the policies and norms of their respective Federal States.

Evaluation procedures for teaching have been introduced by recommendations of the Hochschulrektorenkonferenz, the German Rectors Conference (HRK) and Wissenschaftsrat in order to advance transparency, profile, image and competitiveness of the German HE, to strengthen institutional responsibility and to support HE institutions in introducing measures of systematic quality-promotion.

Through a deal celebrated between the HRK and the KMK an accreditation system was created, the Akkreditierungsrat (Accreditation Council), and the agencies began to be accredited by it. The entire accreditation system had to be designed and built from scratch. Six agencies were accredited with the right to award the Siegel des Akkreditierungsrates, (Quality Certificate of the Akkreditierungsrat) to Bachelor's and Master's degrees of the HEI (ERICHSEN, 2004).

Figure 1 presents a system mapping of the German accreditation system and it details the formal and informal influences arising from the model's decentralization.

Source: adapted from Serrano-Velarde (2008) and Erichsen (2004)

Figure 1 Accreditation System in Germany and formal and informal influences 

Harris Huemmert's (2008) investigation criticizes Germany for not having norms for the selection of evaluators of HE nor for evaluating their performance. The analysis contextualizes the processes of evaluation of HE in Germany, using a case study of an evaluation of educational science in Baden Wurttemberg and explores the way the experts were selected. This study describes the problems encountered and concludes suggesting that there could be room for the introduction of norms in the selection of evaluators in the German evaluation scenario.

In a recent study, Orr and Hovdhaugen (2014) defend extending the access to HE in accordance to European policy agenda by using a "second chance" route which removes the educational attainment criterion in secondary school as a decisive factor in the access to HE. This analysis compares the similar routes approach in Germany, Norway and Sweden, which present different ways, principles and obligations to HEI. Orr and Hovdhaugen (2014) assessed the impact of the second chance opportunity to extend the participation in HE and discussed the contribution of those measures to access inclusive education in the country.

The United Kingdom

Ranked in 10th place in The Global Competitiveness Index 2015-2016, the UK has stood out for its public and private institutions' strong ability to attract talent from abroad, for having some of the best universities in the world and for its high levels of technological adoption (SCHWAB; SALA-I-MARTIN; SAMANS; BLANKE, 2015). According to the Global Competitiveness Index, in the Higher Education and Training category, the country reached the 18th position in the ranking, being placed right behind Germany. It should be noted that in the Quality of Management Schools item, it reached the 3rd position in the overall ranking (SCHWAB, SALA-I-MARTIN, SAMANS, BLANKE, 2015).

Literature on Quality Assurance of HEI in the UK is quite extensive (BARKER, 2007; ELTON, 2000; KOGAN, 1989; STEWART, 2005). The United Kingdom's main characteristic is the universities' vast autonomy, which aren't perceived as being public institutions. The government doesn't intervene directly in the HEI, but indirectly through research councils, which possess an external reference frame for the strategic action in the HEI's (SERRANO-VELARDE, 2008).

In this section, it was taken into consideration the quality and evaluation scenarios in HE in the United Kingdom. The concern about quality and evaluation standards used in HE was highlighted in several approaches (HARVEY, 1995, 2002; ATHANASSOPOULOS, SHALE, 1997; KANJI, MALEK, TAMBI, 1999; BLANDEN, MACHIN, 2004; JACKSON, BOHRER, 2010; MEDLAND, 2014; NAZARKO, ŠAPARAUSKAS, 2014).

Athanassopoulos and Shale (1997) conducted a comparative study, in which they measured the corporate performance of 45 HEI in the UK. Governmental initiatives in this sector gave emphasis to accountability, value for money and cost control, encouraged by the ideas from the New Public Management (NPM) approach and its derivation, in the official weighing scheme followed by the funding agency in order to establish measures and criteria for the allocation of resources to universities. By using the value for money approach, quality is being evaluated in terms of return on investment or expenses. When we use the value for money approach in education, we are emphasizing accountability. In public services, education included, the entities that fund and foment are expected to be accountable. By focusing their study on cost control, Athanassopoulos and Shale (1997) adopted two distinct models for the definition of performance: minimizing costs and maximizing results. Both models must be complementary in order to measure cost efficiency in HEI outcomes. Statistics about activities in the universities reached their full potential when they were used to define broad concepts of performance and goal achievement taking into consideration institutional missions. The study considered six universities in the UK efficient; these demonstrated a satisfactory individual performance in all the tests in accordance with the proposed methodology (ATHANASSOPOULOS; SHALE, 1997). A research about Total Quality Management (TQM) conducted in HE in the United Kingdom by Kanji, Malek, and Tambi (1999) reported that the use of TQM is quite slow with only a few universities using it. These HEI benefitted from the use of TQM (KANJI, 1998) similar to their USA counterparts, when it comes to student performance, improving services, cost reduction, and customer satisfaction. The authors considered how the principles and concepts of TQM have been measured to provide a means to evaluate quality in several aspects of internal processes in HEI. The principles and key concepts of TQM proved to be critical factors of success and mirrored the performance of the institutions. During the study, the Business Excellence Model (KANJI, 1998) demonstrated several advantages and a capacity to resolve insufficiency, which seemed better than other models.

In an external quality monitoring study in HE, Harvey (2002) examined the different external agencies and their modus operandi. His prerogative is that external evaluation legitimizes status quo and is concerned about method, ignoring nature and learning styles. Harvey criticizes the use of statistical indicators as assessment tools, since, according to this author, these have limitations when they are used as quality performance measures. He disputes that in order for self-assessment to work there must be peer review, but if it is bias, the judgement is distorted, seeing that it was based on discrepancies and personal judgement (HARVEY, 2002). Harvey (2002) is quite radical when he proposes that HE monitoring agencies need to be concern about quality and to tackle the implications quality brings to student learning. So that they stop being agencies concerned with accountability and compliance and start raising important questions about improving student learning.

When analyzing the quality and standards in HE in the UK, Jackson and Bohrer (2010) discussed the responsibility of insuring quality in the country. The study highlighted some characteristics of the British system, which is based in autonomous and self-regulating institutions (Figure 2). The analysis questioned the transitional process during the consolidation of the evaluation cycle of 2006-2011. It was also possible to identify the historical role played by the Quality Assurance Agency (QAA) which, at first, was divided into institutional evaluations and courses (programmes) evaluations. The first were conducted through audits by the Higher Education Quality Council (HEQC); the latter were carried out through quality evaluation, using subjective tools established by the Higher Education Funding Council of England (HEFCE).

Source: adapted from Jackson and Bohrer (2010).

Figure 2 Evolution of the evaluation process of the Quality Assurance Agency (QAA) 

During the historical approach, Jackson and Bohrer (2010) also presented the organizations that preceded QAA which established both the systems and process that are currently used to indicate academic standards assurance in the UK. In short, audit processes by QAA are conducted by evaluators who are senior officials appointed by institutions and trained by the agency. They analyze the documentation and meet with staff and students to evaluate the efficiency of the quality systems. Their findings are made available in reports published by QAA. From January 2010 onwards, students joined the audit teams in pilot mode.

More recent studies in the United Kingdom investigated HEI's response capacity, administrators' profile, the proportion and problems created by foreign students, priority and resistance to evaluation and to quality assurance processes in HE in the country (HEMPSALL, 2014; HOGAN, 2014; LUCAS, 2014; MCDONALD, 2014; MEDLAND, 2014).

When studying leadership in HEI, Hempsall (2014) was unable to identify an efficient level of readiness or response capacity one would expect in the era of knowledge in the USA, the UK and Australia. Hogan (2014) examined the administrators in HE in the UK and noticed that the profiles, becoming more flexible. Administrative functions are more diversified. Technological changes have created new roles and opportunities and also the possibility of career development for staff. Foreign students account for an increasing portion of student population in the UK. A study conducted by McDonald (2014) addressed the additional complications brought in by the support to foreign students, with special focus on cultural issues, and made recommendations on how to improve their practices in the United Kingdom.

By emphasizing the role of evaluation in supporting student development during the evaluating processes and practices in HE, Medland (2014) contributes to the discussion about evaluation in HE in the UK, by presenting an overview of the role of evaluation in current curricular change as a way to enhance evaluation. It was analyzed the academic resistance to quality assurance processes in HE and to the evaluation of research work in the UK. Quality processes were seen as corrective technology (Blackmore, 2009) and are an important part of the NPM approach introduced in the sector. The processes were introduced to monitor and control academic work regarding teaching and research in HEI in the UK in the last 20 years. Scholars from specific contexts started to challenge and to resist to discourses and positions (THOMAS; DAVIES, 2005) which were being imposed upon them (LUCAS, 2014).

Methodology

This study is characterized as a descriptive research since it sought to describe the social phenomenon of evaluation and accreditation in HE based on the practice of evaluation agencies in Europe. To support the review of literature concerning quality assurance mechanisms and HE scenarios, from the point of view of evaluation and quality, it was carried out a research in reference databases, based in articles selected from relevant journals from this field of knowledge and relevant impact factor. By analyzing the dimensions discussed in the selected papers from different countries, it was possible to identify the main issues regarding quality and evaluation, identified in studies conducted or applied in HE.

It was chosen a hierarchy diagram to visually represent, in statements, the significant links between the discovered concepts. This is a concept map widely used in knowledge management studies and can be understood as a two-dimensional diagram, whose main function is to display concepts hierarchically organized and the relations between concepts. The connection lines represent the relationship between concepts (MOON,; HOFFMAN; NOVAK, 2011). The computer software used to create the concept map was Cmap Tools Knowledge Modelling Kit. This is a methodological tool which uses assimilation theory to determine what the student knows or found out during literature review (CAÑAS; FORD; COFFEY; REICHHERZER; CARFF, 2000). The recent study by Hagemans, Van Der Meij, and de Jong (2013) reports the experience of students who visualized the concept map and overcame students who only conducted a descriptive analysis. Results show that using concept maps helps improve learning and discover connections between concepts. The use of concept maps to demonstrate concepts and relationships uncovered in the studied countries is justified if we take into consideration the above-mentioned contributions of concept maps.

Discussion

This section presents an overview of the evaluation and accreditation process in Europe: the identified results and the mapping of the European scenario of quality assurance and evaluation in HE in the countries surveyed.

Germany

It was created a two-dimensional synthesis based on the theoretical review findings, which will be demonstrated through a literature review concept map (Figure 3).

Source: created by the authors.

Figure 3 Concept mapping of quality and evaluation in European HE - Germany and United Kington 

The map aims at presenting quality evaluation in HE scenario in Germany and shows the development of evaluation, the increase of private HE in the country, and higher education reforms. It addresses the lack of norms when it comes to the selection of HE evaluators, and policies aiming at expanding higher education in the country, including the establishment of alternative and inclusive routes of access.

Particular attention should be drawn to the resistance and the difficulties found in the country in what concerns the publication of evaluation results by HEI, and likewise the refusal to create a public ranking on the part of governmental authorities and institutions. There is also the matter of the striking differences in the qualification of faculty members internally and between HEI, which is worrisome, and its impact must be analyzed in HE policies. The introduction of new HE policies, with the support of public funding as a way of boosting performance in HEI, is a relevant factor when it comes to quality in HE in the country.

Currently, according to the IIE (2013), Germany received 6% of the total 4,5 million international enrolling students throughout the world, reaching fifth place as a students' host country. This result may confirm the success of the internationalization policies implemented in the country and it explains some of the impacts cited in Table 1. Furthermore, in the technology-oriented universities, the Technische Universitäten and in the smaller colleges, the Fachhochsulen, 40% of the Higher Education Programs can lead to a double degree, including in their curricula a period of study abroad (EPCCE, 2015).

Table 1 Overview of Higher Education in Germany 

Critical Points - Resistance to the publishing policy of university rankings
- Institutional and political resistance to the publication of evaluation results from HEI
- Little funding of HE in the 80s and 90s
- Striking differences in faculty qualification in HEI
- Precarious teaching conditions in HEI; few measures in place for control of quality
- Without specific criteria to conduct internal and internal evaluation
- Serious problems when it came to compare performance between universities.
Implemented improvements - Introduction of new financing systems as a policy to develop HE
- New policies focusing on increasing competitiveness in HE
- Changes to the criteria to access HE used as an incentive - introduction of an alternative route (second chance).
- Policies based on performance indicators in HE
- The 1998 change to Hochschulrahmengesetz, which implemented a reform to HE in Germany
- Establishment of standards for the annual reports on education suggested in the Conference of Rectors and Vice Chancellors of the Universities and HEI
- The foundation of the Centre for Development of Higher Education (Zentrum) which aims to promote quality in HE through internal and external evaluations of faculty performance.
Impacts generated by the changes - Drastic increase in the number of students until 2000, reaching 1.8 million
- Vacancies for students, but the numbers of professors did not sustain this growth
- The ratio professor/student increased from 1/16 to 1/24
- Teaching deregulation, emphasis on performance indicators and incentives (increased resources) based on performance, created by the new Hochschulrahmengesetz (Federal Basic Law of Higher Education).

Source: Produced based on concept mapping of quality and evaluation in HE in Germany (EL HAGE, 1997; HÜFNER, 2003; HARRIS HUEMMERT, 2008; ORR, HOVDHAUGEN, 2014).

Table 1 presents a textual and complementary overview of HE in Germany. It was based in the work of El Hage (1997), Hüfner (2003), Harris Huemmert (2008) and Orr and Hovdhaugen (2014) and highlights the key critical points of HE in Germany, the initiatives proposed and implemented in search of improvements, and the main impacts created by changes in HE.

The following section gives an overview of the finding inherent to quality in HE in the UK.

The United Kingdom

Figure 3 is the result of literature review in the country and shows a two-dimensional diagram which characterizes the situation of the country through a concept map. The mapping intends to present the scenario, in a more visual way, of quality evaluation in HE in the UK.

The key concepts were highlighted through the use of rectangles and directional arrows, which show the connection between concepts, as well as short descriptions along the arrows. Key concepts regarding quality characteristics in HE in the country are presented in Figure 3. It is worth mentioning the well-defined governmental policies of accountability, value for money and cost control. The concern with quality assurance and quality security mechanisms is shown in autonomy characteristics in HE, such as self-regulation and autonomous functioning HEI, based on the policy value for money. This dedication to academic standards assurance, using TQM, achieved results in terms of good performance and became a key factor of success in HEI in the country.

The NPM scenario, in the context of HE, can be understood in the mapping as the cause of the resistance and conflicts in academia, especially the excessive concern with the methods and the monitoring and control of quality rather than the focus on improving learning.

Table 2 presents a textual and complementary overview of HE in United Kingdom and highlights the key critical points, the initiatives proposed and implemented in search of improvements, and the main impacts created by changes in HE.

Table 2 Overview of Higher Education in United Kingdom. 

Critical Points - The need to measure corporate performance of universities.
- Slow progress of TQM in universities and few adoptions of this system.
- External monitoring of quality if HE.
- Autonomous and self-regulating HEI.
Implemented improvements - Establishment of measures and criteria for the allocation of resources to universities with emphasis in NPM, focusing on accountability, value for money and cost control in HE.
- The use of TQM to improve internal processes in the universities using it.
- Criticism regarding skepticism when it comes to accountability and compliance required by evaluation agencies.
- Discussion of the role and the evolution of QAA in the evaluation process of HE.
- Changes to the profile of administrative activities with the creation of new roles and opportunities for professional growth.
Impacts generated by the changes - Activities in universities reached their potential value. Taking into consideration the application of models that minimize costs and maximize results.
- In these HEI that embraced TQM, the business excellence model (Kanji, 1998) has better results than other models of quality management.
- The need to take into consideration relevant issues when it comes to improving students' learning in the external evaluation of the HEI.
- Assessments from quality evaluation agencies (QAA) have evolved and started to include students from HEI in audit teams.
- The increasing number of international students in HE generates the need for new policies to improve practice in HE.
- Academic resistance to the imposition of NPM practices and discourses in the teaching and research environments.

Source: Produced based on Concept Mapping of Quality and Evaluation in HE in UK.

Final Considerations

The European accreditation model suggested by the European Network for Quality Assurance in Higher Education (ENQA) was the point of reference for the mapping of HE in the different countries. The action of the accreditation agencies in HE is paramount when it comes to the control and statistic data collection of teaching realities in these countries.

The study presented the main definitions of quality as well as the tools used to insure quality in HE in Germany and the United Kingdom. It was created a concept mapping of HE in these countries and it was also discussed the characteristics, restrictions and evolution of the processes of accreditation, evaluation and peer review. The main points of action of quality policies and quality agencies in HE, and their current situation of development were demonstrated too.

As for the analysis of HE in Germany, El Hage (1997) highlighted the main positive outcomes and the improvements that emerged after the reform in the German teaching system, with particular attention to the changes carried out in 1998 with the Hochschulrahmengesetz, the new Federal Basic Law of Higher Education in Germany and the implementation of new public policies emphasizing performance indicators in HE (HUFNER, 2003). It is also mentioned the increased competitiveness and the resulting improvement in teaching quality through the increase of offer by the new private HEI. It is worth noting that these changes brought with them an increase in the numbers of students in HE in Germany (ORR; HOVDHAUGHEN, 2014) after the year 2000. However, the downside was that qualified workforce, according to Hufner (2003), did not keep pace with the expansion of teaching, which led to an increase of 33% of the professor-student ratio in German universities, as well as several legal problems concerning regulation and institutional evaluations of the new HEI (HÜFNER, 2003; HARRIS HUEMMERT, 2008).

With reference to HE in the UK, the emphasis on accountability that guided the main discussions and showed that the positive outcomes in HE in this country were due to initiatives, such as value for money and cost control (ATHANASSOPOULOS; SHALE, 1997), concerns with quality assurance mechanisms (KANJI, 1998), and with quality security (KANJI; MALEK; TAMBI, 1999). Mostly, these are characteristics of public policies that ensure autonomy and self-regulation in HE (HARVEY, 2002) in the UK. In this country, the dedication in ensuring that academic standards with resource to TQM achieve good performance results became a crucial success factor for HEI. The NPM scenario, in the context of HE, brought quality processes to light (BLACKMORE, 2009). The more recent studies held in The UK provide a very favorable picture of its development stage, highlighting the response capacity of HEI, the profile of training of administrators in HE, the large proportion of foreign students in the country, the concern in minimizing cultural conflicts, the priority given to evaluation in HE and quality assurance processes in British HEI (HEMPSALL, 2014; HOGAN, 2014; LUCAS, 2014; MCDONALD, 2014; MEDLAND, 2014).

From the point of view of institutional accountability, in Germany and the UK, the lessons learned focused on improvements based in performance in relation to previously established patterns. It was also noted the adoption of accreditation parameters, combining performance indicators, self-assessment, and peer review, applied to courses and programmes and controlled by accreditation agencies of each country. These policies ended up converging in these two countries and meeting Massy's (1997; 2010) precepts.

The quality policies identified in this study tend to conduct their results based on efficiency (control mechanisms) and end up stimulating competition between HEI which may lead, in the case of the UK, to an administrative decentralization, or, in both cases, Germany and the UK, to an emphasis on efficiency, taking into consideration the effectiveness of value for money when allocating resources and the results HEI get on their external evaluations.

The concept mapping suggested in this paper may contribute to compare the efficiency of public policies in HE in the two countries mentioned, considering that the tool currently used by quality evaluation agencies have many common points in the European Union.

Regarding the actions of agencies for accreditation and regulation in HE, although there are similarities in the assurance mechanisms, through educational policies in use guided by ENQA, further study is necessary, focusing exclusively on the actions and outcomes of these agencies, which would complement and explain the scenario mapped in this paper.

Referências

AMARAL, Alberto; ROSA, Maria João. International review of the Portuguese quality assurance system. In: BESO, Anela; BOLLAERT, Lucien; CURVALE, Bruno; TOFT JENSEN, Henrik; HARVEY, Lee; HELLE, Emmi; MAGUIRE, Brian; MIKKOLA, Anne; SURSOCK, Andrée. Implementing and using quality assurance: strategy and practice. A selection of papers from the 2nd European quality assurance Forum. Italy: Università di Roma, 2008. [ Links ]

ATHANASSOPOULOS, Antreas; SHALE, Estelle. Assessing the Comparative Efficiency of Higher Education Institutions in the UK by the Means of Data Envelopment Analysis. Education Economics, v. 5, n. 2, p. 117-134, 1997. Disponível em: <https://doi.org/10.1080/09645299700000011> Acesso em: 12 jan. 2019. [ Links ]

ANDERSON, Garry. Correlational research. In: ANDERSON, Garry; ARSENAULT, Nancy. (Org.). Fundamentals of educational research. London: Falmer Press Teachers Library. 1999. [ Links ]

BARKER, Katharine. The UK Research Assessment Exercise: the evolution of a national research evaluation system. Research evaluation, v. 16, n. 1, p. 3-12, 2007. Disponível em: <https://doi.org/10.3152/095820207X190674> Acesso em: 21 dez. 2019. [ Links ]

BERNER, Heike; RICHTER, Roland. Accreditation of degree programmes in Germany. Quality in Higher Education, v. 7, n. 3, p. 247-257, 2001. Disponível em: <https://doi.org/10.1080/13538320120098122> Acesso em: 12 nov. 2018. [ Links ]

BLACKMORE, Jill. (2009). Academic pedagogies, quality logics and performative universities: evaluating teaching and what students want. Studies in Higher Education, v. 34, n. 8, p. 857-872, 2009. Disponível em: <https://doi.org/10.1080/03075070902898664> Acesso em: 15 nov. 2018. [ Links ]

BLANDEN, Jo; MACHIN, Stephen. Educational Inequality and the Expansion of UK Higher Education. Scottish Journal of Political Economy, v. 51, n. 2, p. 230-249, 2004. Disponível em: <https://doi.org/10.1111/j.0036-9292.2004.00304.x> Acesso em: 1º out. 2018. [ Links ]

BRENNAN, John; SHAH, Tarla. Managing quality in higher education: An international perspective on institutional assessment and change. Buckingham: OECD, 2000. [ Links ]

BRITTINGHAM, Barbara. Accreditation in the United States: how did we get to where we are? New Directions for Higher Education, p. 7-27, 2009. Disponível em: <https://doi.org/10.1002/he.331> Acesso em: 19 sep. 2018. [ Links ]

CAÑAS, Alberto; FORD, Kenneth; COFFEY, John; REICHHERZER, Thomas; CARFF, Roger. Herramientas para construir y compartir modelos de conocimiento basados en mapas conceptuales. Revista de Informática Educativa, v. 13, n. 2, p. 145-158, may-2000. [ Links ]

COSTES, Nathalie; CROZIER, Fiona; CULLEN, Peter; GRIFOLl, Josep. Quality Procedures in the European Higher Education Area and Beyond-Second ENQA Survey. European Association for Quality Assurance in Higher Education (ENQA) occasional papers 14. ENQA. Brussels: Belgium, 2008. Disponível em: <http://eric.ed.gov/?id=ED542058> Acesso em: 07 ago. 2018. [ Links ]

DILL, David. Designing academic audit: Lessons learned in Europe and Asia. Quality in Higher Education, v. 6, n. 3, p. 187-207, 2000. Disponível em: <https://doi.org/10.1080/13538320020005945> Acesso em: 27 nov. 2018. [ Links ]

DILL, David. Capacity building as an instrument of institutional reform: Improving the quality of higher education through academic audits in the UK, New Zealand, Sweden, and Hong Kong. Journal of Comparative Policy Analysis: Research and Practice, v. 2, n. 2, p. 211-234, 2000a. Disponível em: <https://doi.org/10.1080/13876980008412643> Acesso em: 03 fev. 2019. [ Links ]

EATON, Judith. An Overview of US Accreditation-Revised. Council for Higher Education Accreditation. Washington, DC: ERIC, 2012. [ Links ]

EL HAGE, Natalija. Evaluation of Higher Education in Germany. Quality in Higher Education, v. 3, n. 3, p. 225-233, 1997. Disponível em: <https://doi.org/10.1080/1353832970030303> Acesso em: 03 fev. 2019. [ Links ]

ELTON, Lewis. The UK research assessment exercise: unintended consequences. Higher Education Quarterly, v. 54, n. 3, p. 274-283, 2000. Disponível em: <https://doi.org/10.1111/1468-2273.00160> Acesso em: 4 jan. 2019. [ Links ]

EUROPEAN ASSOCIATION FOR QUALITY ASSURANCE IN HIGHER EDUCATION (ENQA). Guidelines for national reviews of ENQA member agencies. ENQA Board. Helsinki: European Association for Quality Assurance in Higher Education, 2006. Disponível em: <http://siupt.uportu.pt/content/files/gqa/ENQA_Guidelines_for_national_reviewsDez2006.pdf> Acesso em: 29 dez. 2018. [ Links ]

EUROPEAN PARLIAMENT'S COMMITTEE ON CULTURE AND EDUCATION (EPCCE). Internationalization of Higher Education. European Union: European Parliament, 2015. [ Links ]

ERICHSEN, H.U. Quality Assurance and Accreditation in Germany. In: DI NAUTA, Primiano; OMAR, Pirjo-Liisa; SCHADE, Angelika; SCHEELE, Jacob (Org.). Accreditation Models in Higher Education − experiences and perspectives. Helsinki: ENQA Workshop Reports 3, 2004. [ Links ]

FANDEL, Günter. On the performance of universities in north rhine-Westphalia, Germany: Government? Redistribution of funds judged using DEA efficiency measures. European Journal of Operational. Research, v. 176, n. 1, p. 521-533, 2007. Disponível em: <https://doi.org/10.1016/j.ejor.2005.06.043> Acesso em: 15 dez. 2018. [ Links ]

HAGEMANS, Mieke; VAN DER MEIJ, Hans; DE JONG, Ton. The effects of a concept map-based support tool on simulation-based inquiry learning. Journal of Educational Psychology, v. 105, n.1, p. 1-24, 2013. Disponível em: <https://doi.org/10.1037/a0029433> Acesso em: 15 ago. 2018. [ Links ]

HARRIS HUEMMERT, Susan. Evaluators of Higher Education in Germany: Are They "Fit For Purpose"? Quality in Higher Education, v. 14, n. 1, p. 55-65, 2008. Disponível em: <https://doi.org/10.1080/13538320802011722> Acesso em: 15 nov. 2018. [ Links ]

HARVEY, Lee. Beyond TQM. Quality in Higher Education, v. 1, n. 2, p. 123-146, 1995. Disponível em: <https://doi.org/10.1080/1353832950010204> Acesso em: 12 dez. 2018. [ Links ]

HARVEY, Lee. Evaluation for What? Teaching in Higher Education, v. 7, n. 3. p. 245-263, 2002. Disponível em: <https://doi.org/10.1080/13562510220144761> Acesso em: 03 ago. 2018. [ Links ]

HARVEY, Lee; WILLIAMS, James. Fifteen Years of Quality in Higher Education. Quality in Higher Education, v. 16, n. 1, p. 3-36, 2010. Disponível em: <https://doi.org/10.1080/13538321003679457> Acesso em: 01 jul. 2018. [ Links ]

HEMPSALL, Kay. Developing leadership in higher education: perspectives from the USA, the UK and Australia. Journal of Higher Education Policy and Management, v. 36, n. 4, p. 383-394, 2014. Disponível em: <https://doi.org/10.1080/1360080X.2014.916468> Acesso em: 25 nov. 2018. [ Links ]

HOGAN, John. Administrators in UK higher education: who, where, what and how much? Perspectives: Policy and Practice in Higher Education, v. 18, n. 3. p. 76-83, 2014. Disponível em: <https://doi.org/10.1080/13603108.2014.930073> Acesso em: 25 nov. 2018. [ Links ]

HIGHER EDUCATION FUNDING COUNCIL OF ENGLAND (HEFCE). Quality Assessment. Consultation Paper. Bristol: HEFCE, 1992. [ Links ]

HIGHER EDUCATION FUNDING COUNCIL OF ENGLAND (HEFCE). Research Assessment Exercise 2008: the outcome. Bristol: HEFCE, 2008. [ Links ]

HIGHER EDUCATION QUALITY COUNCIL (HEQC). Learning from Audit. London: HEQC, 1994. [ Links ]

HÜFNER, Klaus. Governance and Funding of Higher Education in Germany. Higher Education in Europe, v. 28, n. 2, p. 145-163, 2003. Disponível em: <https://doi.org/10.1080/03797720304104> Acesso em: 13 dez. 2018. [ Links ]

INSTITUTE OF INTERNATIONAL EDUCATION (IIE). Project Atlas: trends and global data 2013, 2013. Disponível em: <http://www.iie.org/en/Research-and-Publications/Project-Atlas> Acesso em: 09 dez. 2018. [ Links ]

JACKSON, Stephen; BOHRER, Janet. Quality assurance in higher education: Recent developments in the United Kingdom. Research in Comparative and International Education, v. 5, n. 1, p. 77-87, 2010. Disponível em: <https://doi.org/10.2304/rcie.2010.5.1.77> Acesso em: 19 jan. 2019. [ Links ]

KANJI, Gopal. Measurement of business excellence. Total Quality Management, v. 9, n. 7, p. 633-643, 1998. Disponível em: <https://doi.org/10.1080/0954412988325> Acesso em: 21 jan. 2019. [ Links ]

KANJI, Gopal; MALEK, Abdul; TAMBI, Bin A. Total quality management in UK higher education institutions. Total Quality Management, v. 10, n. 1, p.129-153, 1999. Disponível em: <https://doi.org/10.1080/0954412998126> Acesso em: 04 jan. 2019. [ Links ]

KASTELLIZ, Dietlinde; MITTERAUER, Barbara. (2014). Quality Audit in the European Higher Education Area: a Comparison of Approaches. Vienna: Facultas Verlags - und Buchhandels, 2014. Disponível em: <http://www.pedocs.de/frontdoor.php?source_opus=12088> Acesso em: 17 jan. 2019. [ Links ]

KELLEY, Craig; TONG, Pingsheng; CHOI, Beom-Joom. A review of assessment of student learning programs at AACSB schools: A dean's perspective. Journal of Education for Business, v. 85, n. 5, p. 299-306, 2010. Disponível em: <https://doi.org/10.1080/08832320903449519> Acesso em: 24 jan. 2019. [ Links ]

KEMPKES, Gerhard; POHL, Carsten. The efficiency of German universities: Some evidence from nonparametric and parametric methods. Applied Economics, v. 42, n. 16, p. 2063-2079, 2010. Disponível em: <https://doi.org/10.1080/00036840701765361> Acesso em: 09 fev. 2019. [ Links ]

KNUPFER, Nancy; MCLELLAN, Hilary. Descriptive research for educational communications and technology. New York: Macmillan USA, 1996. [ Links ]

KOGAN, Maurice. Evaluating Higher Education. Papers from the International Journal of Institutional Management in Higher Education. Centre for Educational Research and Innovation. Organisation for Economic Co-operation and Development (OECD). Bristol/PA: Taylor & Francis, 1989. [ Links ]

LUCAS, Lisa. Academic resistance to quality assurance processes in higher education in the UK. Policy and Society, v. 33, n. 3, p. 215-224, 2014. Disponível em: <https://doi.org/10.1016/j.polsoc.2014.09.006> Acesso em: 19 jan. 2019. [ Links ]

MASSY, William. Teaching and Learning Quality‐process Review: the Hong Kong programme. Quality in Higher Education, v. 3, n. 3, p. 249-262, 1997. Disponível em: <https://www.tandfonline.com/doi/abs/10.1080/1353832970030305> Acesso em: 01 nov. 2018. [ Links ]

MASSY, William. Academic audit for accountability and improvement. Achieving accountability in higher education: Balancing public, academic and market demands. San Francisco: Jossey-Bass, 2005. [ Links ]

MASSY, William. Education quality audit as applied in Hong Kong. Public Policy for Academic Quality, v. 30, n. 1, p. 203-225, 2010. Disponível em: <https://doi.org/10.1007/978-90-481-3754-1_11> Acesso em: 22 jan. 2019. [ Links ]

MCDONALD, Ian. Supporting international students in UK higher education institutions. Perspectives: Policy and Practice in Higher Education, v.18, n. 2, p. 62-65, 2014. Disponível em: <https://doi.org/10.1080/13603108.2014.909900> Acesso em: 22 jan. 2019. [ Links ]

MEADE, Philip; WOODHOUSE, David. Evaluating the effectiveness of the New Zealand Academic Audit Unit: Review and outcomes. Quality in Higher Education, v. 6, n. 1, p. 19-29, 2000. Disponível em: <https://doi.org/10.1080/13538320050001045> Acesso em: 23 jan. 2019. [ Links ]

MEDLAND, Emma. Assessment in higher education: drivers, barriers and directions for change in the UK. Assessment & Evaluation in Higher Education, v. 18. n. 2, p. 81-96, 2014. Disponível em: <https://doi.org/10.1080/02602938.2014.982072> Acesso em: 27 jan. 2019. [ Links ]

MELTZOFF, Julian. Critical thinking about research: Psychology and related fields. Washington DC: American Psychological Association, 1998. [ Links ]

MERTENS, Donna. Research methods in education and psychology: Integrating diversity with quantitative & qualitative approaches. London: Sage Publications, 1998. [ Links ]

MONTEIRO, Renato; PEREIRA, Cleber; SOUZA, Marcos. A necessidade da informação gerencial nas IES Públicas da Europa: breve abordagem pela realidade de Portugal. Revista Catarinense da Ciência Contábil, v. 14, n. 42, p. 9-24, 2015. Disponível em: <https://doi.org/10.16930/2237-7662/rccc.v14n42p9-24> Acesso em: 02 fev. 2019. [ Links ]

MOON, Brian; HOFFMAN, Robert; NOVAK, Joseph. Applied concept mapping: capturing, analysing, and organizing knowledge. Boca Ratom, Flórida: CRC Press Taylor & Francis Group, 2011. [ Links ]

NAZARKO, Joanicjusz; ŠAPARAUSKAS, Jonas. Application of DEA method in efficiency evaluation of public higher education institutions. Technological and Economic Development of Economy, v. 20, n. 1, p. 25-44, 2014. Disponível em: <https://doi.org/10.3846/20294913.2014.837116> Acesso em: 06 fev. 2019. [ Links ]

ORR, Dominic; HOVDHAUGEN, Elisabeth. "Second chance" routes into higher education: Sweden, Norway and Germany compared. International Journal of Lifelong Education, v. 33, n. 1, p. 45-61, 2014. Disponível em: <https://doi.org/10.1080/02601370.2013.873212> Acesso em: 06 fev. 2019. [ Links ]

PRINGLE, Charles; MICHEL, Mitri. Assessment Practices in AACSB-Accredited Business Schools. Journal of Education for Business, v. 82, n. 4, p. 202-211, 2007. Disponível em: <https://doi.org/10.3200/JOEB.82.4.202-211> Acesso em: 19 nov. 2018. [ Links ]

RAUHVARGERS, Andrejs; DEANE, Cynthia; PAWELS, Wilfried. Bologna Process Stocktaking Report 2009. Report from working groups appointed by the Bologna Followup Group to the Ministerial Conference in Leuven/Louvain-la-Neuve: Belgium, 2009. [ Links ]

SANTOS, Sérgio Machado dos. Análise comparativa dos processos europeus para a avaliação e certificação de sistemas internos de garantia da qualidade. Lisboa, Portugal: A3ES Readings, 2011. [ Links ]

SCHWAB, Klaus; SALA-IARTIN, Xavier; SAMANS, Richard; BLANKE, Jennifer. The global competitiveness Report 2015-2016: Highlights. In: World Economic Forum, 2015. Disponível em: <http://www3.weforum.org/docs/gcr/2015-<2016/Global_Competitiveness_Report_2015-2016.pdf> Acesso em: 08 dez. 2018. [ Links ]

SILVA-TRIVINO, Moises; RAMIREZ-GATICA, Soledad. External review teams training in Central America. Quality in Higher Education, v. 10, n. 3, p. 261-265, 2004. Disponível em: <https://doi.org/10.1080/1353832042000299540> Acesso em: 11 jan. 2019. [ Links ]

STEWART, Jim. The UK research assessment exercise. Human Resource Development International, v. 8, n. 4, p. 489-494, 2005. Disponível em: <https://doi.org/10.1080/13678860500355616> Acesso em: 14 nov. 2018. [ Links ]

THOMAS, Robin; DAVIES, Annette. Theorizing the micro-politics of resistance: New Public Management and Managerial Identities in the UK Public Services. Organization Studies, v. 26, n. 5, p. 683-706, 2005. Disponível em: <https://doi.org/10.1177/0170840605051821> Acesso em: 29 dez. 2018. [ Links ]

WARNING, Susanne. Performance differences in German higher education: empirical analysis of strategic group. Review of Industrial Organization, v. 24, n. 4, p. 393-408, 2004. Disponível em: <https://doi.org/10.1023/b:reio.0000037538.48594.2c> Acesso em: 19 jan. 2019. [ Links ]

Received: March 15, 2019; Accepted: April 15, 2019

Doutorando Cléber Augusto Pereira Universidade do Minho (Portugal) Escola de Economia e Gestão Centro de Investigação em Ciência Política (CICP) ORCID ID: http://orcid.org/0000-0002-7704-2343 Email: kcleber@gmail.com

Prof. Ph.D. Joaquim Filipe Ferraz Esteves de Araújo Universidade do Minho (Portugal) Programa de Doutorado em Ciências da Administração Escola de Economia e Gestão Centro de Investigação em Ciência Política (CICP) ORCID ID: http://orcid.org/0000-0001-8531-6036 Email: jfilipe@eeg.uminho.pt

Profa. Ph.D. Maria de Lourdes Machado-Taylor Centro de Investigação em Políticas do Ensino Superior (CIPES-Portugal) Agência de Avaliação e Acreditação do Ensino Superior (A3ES-Portugal) ORCID ID: http://orcid.org/0000-0003-4774-2397 Email: lmachado@cipes.up.pt

Creative Commons License Este é um artigo publicado em acesso aberto (Open Access) sob a licença Creative Commons Attribution NonCommercial, que permite uso, distribuição e reprodução em qualquer meio, sem restrições desde que sem fins comerciais e que o trabalho original seja corretamente citado.