FCT and OECD hold meeting on evaluating the impact of public research

Effective evaluations of funding programs and instruments are widely recognized as key to informing the preparation of public policies for science. The relevance of effective impact assessment is especially pertinent in times of increasing budgetary constraints. However, in OECD member states, the panorama of public research impact assessment (carried out in public and semi-public institutions) is diverse and inconsistent, so there is a clear need for reflection on the different practices in place, on implementation and on the effectiveness of impact assessment as a support for policy decisions.

This was the main objective of the workshop "Assessing the Impacts of Public Research Systems", which took place in Lisbon at the end of April. In a joint FCT-OECD organization, researchers, professionals from public research systems and policymakers from 14 OECD countries, as well as representatives from the European Commission, Science Europe and the OECD itself, met to reflect on and learn from the experiences of the various countries in assessing the impact of public research systems. The results of the workshop pave the way for the definition of priorities, indicators and best practices that can serve as a foundation for transnational analysis.

The international panorama of impact evaluation is indeed disparate: in several countries ex-post evaluation is mandatory, when programmes come to an end; some countries also require an ex-ante impact analysis, which serves to support programmes and policies; a small group of countries go further, implementing long-term evaluations, which make it possible to measure the economic and social impact of programmes or organizations. Portugal is in the group (made up mostly of southern European countries) that, as a general rule, implement evaluation exercises as a requirement for the use of European Structural Funds. Clearly, these countries, and others in the OECD, can benefit from the knowledge of countries with more experience in this area, and thus develop not only skills but also a solid knowledge base (namely by broadening the scientific community).

Despite the existing disparities, the scope of evaluations has been increasing since the first studies were carried out in the 60s. Any evaluation of public research systems today covers individuals and their academic-professional paths, organizations and institutions, networks and, more recently, ecosystems and policy-mixes. This comprehensiveness implies the simultaneous and rigorous analysis of several instruments and, in particular, the often complex interactions between instruments.  

The challenges increase when we move on to the level of indicators and methodologies to be applied in transnational analyses. In this sense, the Lisbon workshop showed that sharing knowledge, identifying common issues and concerns, and reflecting on shared experiences and results are essential steps for the success of the process.

Participants in the workshop identified that the different challenges associated with impact assessment studies at regional, national or international level may require different methodological approaches. They also stressed the importance of incorporating impact assessment studies into regular institutional practice in the policy-making process. The issue of excellence in impact assessment was also addressed, recognizing that only studies carried out according to the highest international quality criteria effectively contribute to optimizing the design of public policies.

The results of the workshop will be directly used in the Knowledge Triangle project of the Working Party on Innovation Policies (TIP) of the OECD's Committee on Scientific and Technological Policy (CSTP). This new project aims to analyze the various aspects of the involvement of universities, research centers and the business community in promoting innovation. Some of the issues that the project will address are: incentives for researchers and other actors in the system, the role of governments as coordinators and agents in the system, the identification of new forms of collaboration between universities and society, and the policies that best harmonize the different roles (current and new) of research, education and innovation in the Knowledge Triangle. Luisa Henriques (FCT) and Ricardo Mamede (ISCTE) are members of the project's Steering Group.

Impact assessment is one of the four modules of the Knowledge Triangle project, the other three being (1) Higher education institutions in the Knowledge Triangle; (2) New funding, cooperation and governance regimes and (3) Place-based policies in support of the Knowledge Triangle.