FCT

R&D Institutions

Resultado da avaliação 2007 na área de Ciências e Politicas da Educação

Unidade de I&D

Centro de Investigação de Políticas do Ensino Superior - CIPES [EDU-Centro-Coimbra-757] visitada em 09/01/2008

Classificação: Excellent

Reapreciação

Na sequência da publicação em Dezembro de 2008 da classificação que lhe foi atribuída pela FCT sob recomendação do painel de avaliação, esta unidade apresentou reclamação e solicitou à FCT uma nova avaliação. Embora não estivessem reunidas as condições necessárias, expressas no ponto 4 do art. 9º do Regulamento do Financiamento Plurianual, a FCT decidiu promover nova avaliação por um painel distinto, uma vez que das 15 unidades da área científica de Ciências e Políticas da Educação para além desta unidade outras 11 apresentaram igualmente reclamação e solicitaram nova avaliação. A classificação acima, que já foi superiormente homologada, é a que resultou desta nova avaliação, em que das 12 unidades envolvidas 2 viram a sua classificação subir de Muito Bom para Excelente, 1 viu a sua classificação subir de Bom para Muito Bom, e 2 viram a sua classificação subir de Regular para Bom.

Os comentários do painel original e os respectivos comentários da unidade são apresentados a seguir a esta secção relativa à reapreciação.

Comentários do Painel de reapreciação

This is a very strong Unit, the strongest amongst all of those we reviewed. We agreed with many of the comments made by the Evaluation Panel about individual groups but not with their overall judgment. For example, in their comments they state that this is ‘one of the most sought after research groups for collaboration within the higher education research field in Europe’. Group 1383 in particular was described as making ‘a remarkable contribution to the field’. The other two groups were also seen as very strong internationally. We would therefore judge that overall the Unit meets the criteria for Grade 5 ‘excellent’ (‘one or more group carrying out internationally recognized outstanding research … while most others did high quality international research’).

One particular comment made by the Evaluation Panel that might explain their lower grade concerns the Unit’s apparent weakness in terms of ‘feasibility’. Feasibility is defined in FCT criteria as ‘the capacity of the group of transforming interesting plans into practical projects that are relevant at the international level’. We would interpret that as the ability to turn research ideas and plans into research projects that are relevant internationally – something that this Unit is exceptionally good at. The Evaluation Panel seemed to interpret this as concerning ‘practical impact’. They comment that, though they heard about many examples of practical impact during their site visit, these were not listed in their original submission. In our view, this is a misinterpretation. ‘Impact’ comes under ‘relevance’ which includes ‘scientific, technical and socio-economic impact’ where again this Unit scores very highly. We agreed with the Evaluation Panel that further development was needed in relation to PhDs but we judged this not to be a sufficient weakness to change our judgment that overall this was an outstanding Unit that clearly justified the grade of ‘excellent’.

Comentários do painel de avaliação
Sobre a unidade
CIPES is both the best higher education research group in Portugal and one of the most sought after groups for collaboration within the higher education research field in Europe. Its list of visitors and collaborators demonstrates the international respect it commands. The Research Assessment format is not helpful to the Centre in three particular ways. First, it has encouraged the centre to present itself as three separate research groups whereas a distinguishing feature of higher education research is the extent to which it is intellectually integrated so, for example, institutional/organisational research (one of the groups) cannot be untangled from consideration of markets, resources and system level developments (topics covered in other groups). Second, one of the criteria “Feasibility” is defined as “reflecting on the capacity of the group of transforming interesting plans into practical projects that are relevant at the international level” but on a strict interpretation only one of the groups, where several publications relate specifically to planning and strategy, could be said to meet this criteria. It is of the nature of the work the Centre carries out, for example, on the Bologna Process, that while it contributes to ideas on a Europe-wide basis, it cannot “transform plans into practical projects”. Perversely, when we met the Centre we heard orally of many instances where the Portuguese Ministry had turned to the Centre for advice and assistance in policy formulation which were not represented in publications. Thirdly, the heading “Productivity” which we have assumed includes quality allows no refinement of marking between 4 and the absolute hallmark of excellence, 5. By comparison with the best work world wide the Centre’s work would not qualify for 5 but much of it—the book series and some articles—qualify for some indication that it rates a ‘good ‘4.
The panel recommends that:
1. The Centre should further invest on its ability to create an active collaborative research endeavour from a group of staff drawn from different disciplinary backgrounds and from different departmental bases in different universities. Here the leadership of Professor Amaral has been of paramount importance.
2. More single authored books should be published in the future.
3. Since the very substantial published outputs from the Centre could be criticised as being too diffuse, there is a need for narrowing down the thematic focus of the research conducted in the Center.
Sobre os grupos de investigação
Institutional and Organisational Analysis (IOA) [RG-EDU-Centro-Coimbra-757-1386]
The Institutional and Operational Analysis (IOA) RG is the largest RG (11 main researchers). It mainly focuses on issues of strategic planning, issues of institutional governmentality in HE and managerial aspects of the milieu (in Bourdieu’s & Alheit’s terms) both at national and international level (adopting a comparative perspective). It performs fine in terms of international research activities and publications, while its achievements are combined to high visibility and inter-disciplinarity. Additionally the evaluated RG has the proper Human Recourses in order to carry out the planned research, even though one can wonder whether a probable merging between IOA and SLP (since they operate in a thematic interface and are constituted by, more or less, the same researchers) could increase the output and mobilise the existing resources in a fully operational manner/ mode in order to contextualize their research within the public policy complex.
Regarding training, IOA achieves below average (only 1 PhD accomplished), despite the fact that it is the largest group.

It is rather inequitable to seek to measure a group which covers such a wide area. Inevitably the publications are spread similarly and do not constitute a coherent theme. They include two very good publications by Graca (2005) and by Taylor, Amaral and Machado (2007) but some of the others are less good. Too many publications are jointly authored and some single authored books would strengthen the groups reputation as would more PhD students.
Resources, Allocation and Human Capital (RAHC) [RG-EDU-Centro-Coimbra-757-1393]
The Resources, Allocation and Human Capital (RAHC) is a 7 researchers’ Research Group, emphasizing in a very intriguing topic, namely the economic aspects and facets of Higher educational policy and Higher Education Institutes’ performance and impact assessment (within the context of a production process approach). Targeted research adopted to the current relevant international trends, well established methodological strategy, contextually embedded approaches, obvious operational capacity and an excellent publications record, highlight an absolutely capable group (even though it’s numerically limited), operating in a new research environment. Only one PhD completed.

This is an important theme; Amaral's own work on markets in higher education is internationally respected and has considerable relevance. The two publications: Teixeira, Jongbloed, Dill, and Amaral (2004) and Teixera, Johnstone, Rosa and Vossensteyn (2006) are highly regarded and involve major authorities from outside the group. On the other hand not all the work listed is of such relevance and quality. Again there is a lack of single authored work of book length. This is an area which ought to attract PhD students and perhaps will do in time.
System Level Policies (SLP) [RG-EDU-Centro-Coimbra-757-1383]
CIPES in general: a final comment
Undoubtedly CIPES is an exceptional case both in terms of international productivity- impact and human resources (including academics with remarkable international reputation such as Guy Neave). It carries out an internationally recognized, high quality research and has a remarkable contribution in the field. To sum up it performs remarkably in the 3 first criteria, and only the performance in the training one is below average (clearly the focus of CIPES isn’t on training).
We could just recommend CIPES to develop its training domain (more PhDs), in order to align its research expertise with young researchers orientations and professional trajectories. We should also stress that SLP and IOA could merge into one group, due to the dynamic interface of their research fields and the existing “overlap” of the staff (apart from Dr Ripes da Rosa and Pires da Fosenca, the rest of the main researchers of the SLP participate IOA, while both the RGs are coordinated by the same researcher). The third RG, constituting CIPES, could keep its operational autonomy, because of its research profile and the subsequent orientation. We just recommend to reinforce-increase its personnel (probably recruiting more experts in HRM, New Public Management and Performance Measurement).
The System Level Policies (SLP) RG (8 main researchers) has a remarkable qualitative and quantitative output (especially in terms of international publications), focuses on crucial research topics (such as the Bologna Process, TQM issues, QA procedures in HE, the implementation of the Open Method of Coordination). Its contribution and impact in the relevant epistemic community is obvious, while its operational capacity reflects on the sufficient record of Conferences organised by SLP (in cooperation with the rest of the RGs). In terms of training, its achievements are average (2 PhDs accomplished). On these grounds the proposed rating is the abovementioned.

The group concept is rather artificial in this case. The publications vary from excellent to relatively routine. However they include some important pieces.Too many publications are joint and there is a lack of single authored books.The group contains two of the major figures in the field and over time can be expected to produce work of great excellence.There needs to be a build up of PhD students for the group to meet its ambitions.

Comentários da unidade

O CIPES discorda do resultado da avaliação que enferma de erros conceptuais, metodológicos e processuais graves, como reconhecido pelos avaliadores: “The Research Assessment format is not helpful to the Centre...
O CIPES chamou a atenção da Coordenação para o formato da avaliação ser mais apropriado para as “hard sciences”, não fazendo sentido a divisão em linhas de investigação. Os avaliadores escrevem: encouraged the centre to present itself as three separate research groups whereas a distinguishing feature of higher education research is the extent to which it is intellectually integrated... Assim o CIPES foi penalizado pela deficiência do instrumento de avaliação.
Segundo os avaliadores é negativa a definição de “Feasibility”: “reflecting on the capacity of the group of transforming interesting plans into practical projects that are relevant...” O modelo privilegia a aplicabilidade em detrimento da busca livre do conhecimento, reforçando o “bias” nas hard sciences. Vá lá que não pediram patentes. A Coordenação devia ler “What are universities for?”, League of European Research Universities, 2008.
Apesar dos protestos, o CIPES foi incluído nas Ciências da Educação o que é um rematado disparate. Os avaliadores, sem ninguém qualificado nas políticas do ensino superior, teve a honestidade de concluir não ter capacidade para a avaliação.
Esta acabou por ser feita numa segunda visita do Presidente da Comissão e um novo perito. A avaliação é, no essencial, a opinião de um avaliador único que não participou na visita aos outros centros, o que é contra todas as regras.
Algumas observações dos avaliadores estão erradas. Declaram não ter visto trabalhos sobre o sistema português (aplicabilidade) o que só pode ser explicado por nenhum deles falar português (à atenção da Coordenação) – não se esperarão relatórios em inglês para o Ministério ou o CNE!!!. A existência de trabalho aplicado sobre a implementação de Bolonha revela desconhecimento sobre Portugal onde a implementação se iniciou em 2006, pelo que dificilmente já haverá estudos (os primeiros artigos – Education for Chemical Engineers e Higher Education são do final de 2008). Ironicamente, um artigo “não aplicado” do Cipes sobre Bolonha foi usado no Parlamento Inglês... (4th report, Select Committee on Education and Skills).