Effective impact assessment of public programmes and funding schemes plays an important role in informing policy decisions on public research. All the more so in contexts of increasingly tight research budgets. However, across OECD-member countries, evaluation of the impact of public research is varied and uneven; thus there is a clear need to reflect on the different practices that are in place, on their uptake and on their effectiveness in supporting policymaking.
This was the primary goal of the joint FCT-OECD workshop “Assessing the Impacts of Public Research Systems”, held in Lisbon at the end of April. Practitioners, academics and policy-makers from 14 OECD-member countries, as well as representatives of the European Commission, Science Europe and the OECD, met to reflect on and learn from national experiences in impact assessment of public research. The outcomes of this exploratory workshop open the way to setting up priorities, indicators and best practices for cross-country analyses.
Indeed, the international evaluation landscape is speckled: ex-post evaluation, upon termination of a programme, is mandatory in several countries; some also require ex-ante impact assessment to be carried out, to support programme and policy design; a few countries go as far as to carry out longer-term evaluation, aiming at measuring economic and other social impacts of programmes or organisations. Portugal falls into the group of countries (mainly Southern European) that carry out evaluation activities mostly as a requirement for use of European Structural Funds. These, as well as other OECD member states, stand to benefit greatly from more experienced countries, when working to develop greater know-how and a solid knowledge base (namely via enlargement of the research community).
Common to all types of evaluation is the scope, which has widened since the early days of research evaluation, in the 1960s. It now encompasses individuals and their trajectories, organisations and institutions, networks, and more recently eco-systems and policy-mixes. This wider scope requires careful exploration of multiple instruments and, even more relevant, of often-complex interactions between instruments.
The challenges increase manifold when extending indicators and methodologies to cross-country analyses. Thus, the workshop underscored how sharing of knowledge, identifying common policy questions and concerns, and reflecting on lessons learned from shared experiences and results are crucial first steps in setting up cross-country analyses.
In particular, participants in the workshop discussed how impact assessment studies addressing different regional, national or international levels face different challenges and may eventually require distinct methodological approaches. The importance of incorporating impact assessment studies into the regular institutional practice of policy making was also emphasised. The issue of excellence in impact assessment studies was also addressed during the workshop: only those studies meeting the highest standards of quality are valuable in optimising the design of policy instruments.
The workshop feeds directly into the Knowledge Triangle project of the Technology and Innovation Policy working group of the OECD Committee for Science and Technology Policy (CSTP). This new project will focus on several issues relating to the roles of universities, of research centres and of industry in promoting innovation. These include incentives for researchers and other actors, the role of governments as both coordinators and actors, identifying new channels for collaboration between university and society, as well as policies to link the different roles (existing and new) for research, education and innovation in the Knowledge Triangle. Luisa Henriques (FCT) and Ricardo Mamede (ISCTE) are members of the project’s Steering Group.
Evaluation and impact assessment is one of the four modules of the Knowledge Triangle project. The other modules are (1) Higher education institutions (HEIs) in the Knowledge Triangle (CSTP); (2) New financing, co-operation and governance arrangements and (3) Place-based policies in support of the Knowledge Triangle.