FCT and OECD promote meeting on the evaluation of the impact of public research

Effective evaluations of funding programs and instruments are widely recognized as fundamental to informing the preparation of public policies for science. The relevance of effective impact assessment is particularly pertinent in times of increasing budgetary constraints. However, in OECD member states, the landscape of public research impact assessment (carried out in public and semi-public institutions) is diverse and inconsistent, so there is a clear need to reflect on the different practices in place, on implementation, and on the effectiveness of impact assessment as a basis for policy decisions.

This was the main objective of the workshop “Assessing the Impacts of Public Research Systems,”which took place in Lisbon at the end of April. In a joint FCT-OECD organization, researchers, professionals from public research systems, and policy makers from 14 OECD countries, as well as representatives from the European Commission, Science Europe, and the OECD itself, came together to reflect on and learn from the experiences of various countries in assessing the impact of public research systems. The results of the workshop pave the way for defining priorities, indicators, and best practices that can serve as a basis for cross-national analysis.

The international landscape of impact assessment is indeed diverse: in several countries, ex-post evaluation is mandatory when programs end; some countries also require ex-ante impact analysis, which supports programs and policies; a small group of countries goes further, implementing long-term evaluations that measure the economic and social impact of programs or organizations. Portugal is in the group (made up mainly of southern European countries) that, as a general rule, implements evaluation exercises as a requirement for the use of European Structural Funds. Clearly, these countries, and others in the OECD, can benefit from the knowledge of countries with more experience in this area, and thus develop not only skills but also a solid knowledge base (notably through the expansion of the scientific community).

Despite existing disparities, the scope of evaluations has been increasing since the first studies were conducted in the 1960s. Any evaluation of public research systems today covers individuals and their academic and professional careers, organizations and institutions, networks, and, more recently, ecosystems and policy mixes. This scope requires the simultaneous and rigorous analysis of various instruments and, in particular, the often complex interactions between them.  

The challenges increase when moving on to the level of indicators and methodologies to be applied in transnational analyses. In this regard, the Lisbon workshop highlighted that knowledge sharing, identifying common issues and concerns, and reflecting on shared experiences and results are essential steps for the success of the process.

Workshop participants identified that different challenges and impact assessment studies at regional, national, or international levels may require different methodological approaches. They also stressed the importance of incorporating impact assessment studies into regular institutional policy-making practice. The issue of excellence in impact assessment was also addressed, recognizing that only studies conducted according to the highest international quality standards contribute effectively to optimizing public policy design.

The results of the workshop will be directly used in the Knowledge Triangle project of the OECD's Committee for Scientific and Technological Policy (CSTP) Task Force on Innovation Policy (TIP). This new project aims to analyze the various aspects of the involvement of universities, research centers, and the business community in promoting innovation. Some of the issues that the project will address are: incentives for researchers and other actors in the system, the role of governments as coordinators and agents in the system, the identification of new forms of collaboration between universities and society, and policies that best harmonize the different (current and new) functions of research, education, and innovation in the Knowledge Triangle. Luisa Henriques (FCT) and Ricardo Mamede (ISCTE) are members of the project's Steering Group.

Impact assessment is one of the four modules of the Knowledge Triangle project, the other three being (1) Higher education institutions in the Knowledge Triangle; (2) New financing, cooperation, and governance regimes; and (3) Place-based policies in support of the Knowledge Triangle.