RI-PATHS State of Play
The RI-PATHS team has concluded a highly interactive and participatory phase of the project which was dedicated to sourcing input from the Research Infrastructure (RI) community on key building blocks for developing a comprehensive framework on socio-economic impact. In total, nine participatory workshops have been convened helping the RI-PATHS team to identify the key areas for impact measurement, to disentangle main impact pathways for various types of RIs, to structure draft lists of suitable indicators and to assess the applicability of a range of measurement methods. The results of these interactive events are summarised in two reports: D4.1 Concept note on modular impact assessment framework and D4.2 Results of the participatory workshops (available end July 2019).
As the next step, the RI-PATHS team will refine the conceptual framework and discuss it with the project Advisory Board in September 2019. The RI community, policy makers and other interested stakeholders are warmly invited to a validation workshop of the RI-PATHS impact assessment framework on 22 October 2019 in Brussels (Fraunhofer EU Office, Rue Royale 94).
High-level Expert Groups on Research Infrastructures
The RI-PATHS coordinator EFIS Centre is engaged in two High-Level Expert Groups on Research Infrastructures that are relevant to the area of impact assessment.
1. ESFRI AD-HOC WORKING GROUP
In May 2018, the Competitiveness Council adopted conclusions on Accelerating knowledge circulation in the EU which invites Member States and the Commission within the framework of ESFRI "to develop a common approach for monitoring of their Research Infrastructures (RIs) performance and invited the Pan-European RIs, on a voluntary basis, to include it in their governance and explore options to support this through the use of Key Performance Indicators”. ESFRI was asked to implement this mandate and set up an ad hoc working group (WG) whose Terms of Reference (ToRs) define its objective as follows:
Alasdair Reid, Policy Director of EFIS Centre and coordinator of RI-PATHS, was invited to join an ad-hoc working group on monitoring and KPIs to develop a suitable framework that could be applied to the ESFRI Landmarks. The working group met several time in the first semester of 2019 and developed a proposal (including detailed summary sheets for a proposed set of 20 KPIs) that will be presented at an ESFRI Validation Workshop on "Monitoring of Research Infrastructures Methodology and Key Performance Indicators” in Brussels on 3 July 2019. The work of RI-PATHS on indicators for socio-economic impact assessment (IA) was taken into account by the ad hoc working group and the need for RIs to monitor a core set of KPIs over time that allows the development of an evidence base for subsequent IAs was underlined.
2. EUROPEAN COMMISSION HIGH-LEVEL EXPERT GROUP
In January 2019, the European Commission DG Research and Innovation (DG RTD) set up a High-Level Expert Group (HLEG) to assess the progress of ESFRI and Other World Class Research Infrastructures towards implementation and long-term sustainability. Jelena Angelis, Research Director of EFIS Centre, was invited to join the HLEG together with another four independent experts from France, Italy, Portugal and the United Kingdom.
First results of the work will be ready in July 2019 but the whole work will be concluded in December 2019.
Second round of participatory workshops
Between 8th and 10th May 2019, the second round of RI-PATHS workshops took place at the premises of DESY and ALBA as well as close to the ELIXIR campus in Whittlesford. The main aim of this second round was to engage the RI community more in-depth in the identification of suitable indicators and metrics that can in practice serve to measure the main aspects and impact pathways that were identified as relevant and feasible in the first round of workshops. Hence, the workshops focused on sourcing consensus on the selection of relevant indicators and discussions on data gathering routines that could be deployed in future IA exercises. The workshops were structured keeping in mind the objective of exploring more in-depth each of the three main domains of impact assessment identified
- Straightforward impacts to be robustly captured in economic analysis
- Non-quantifiable impacts captured by actor-based analysis,
- Complex network effects captured through exploratory approaches.
Each workshop started with a short presentation of the project and its key milestones, before the consortium team proposed a structure of pathways and a potential core indicator system based on outcomes from T2.4. Subsequently, the workshop participants were asked to scrutinise and develop further the proposed framework with a view to both relevance and feasibility. As a follow up, a survey was sent to all participants (as well as the broader RI-PATHS community) who were asked to review and comment on long-lists of pathways and indicators. Based on this detailed engagement, the workshops resulted in a corroboration (and in part readjustment) of assumptions that will shape both the project's guiding model and the more detailed indicator system to be attached to it.
An additional participatory workshop was held in conjunction with the final event of the MERIL-2 project on 24th June in Lisbon. The discussions focused on topics regarding: 1) commonalities and differences in IA needs between different types and domains of RIs, in particular distinctions between natural science RIs and RIs serving social sciences and humanities; 2) resource needs to carry out IA activities; as well as 3) usage and openness of impact related data.
In summary, the workshops resulted in several central findings:
First, a majority of research infrastructures agree that societal and human capital impact can be as important as economic - and in part even scientific - impact. In particular, distributed research infrastructures see societal impact on equal terms with their impact in the pure scientific domain. At least within the RI community built around RI-PATHS project it is undeniably confirmed that an exclusively scientific orientations constitute the exception rather than the rule, even among single site, hard science infrastructures.
Figure 1: Assessment of Relevance of different impact categories (mostly single-sited, "hard science" infrastructures)
Figure 2: Assessment of Relevance of different impact categories (mostly distributed, partially "soft science" infrastructures)
Second, however, the majority of research infrastructure managers are rather pessimistic about the means available. Many are aware of the opportunities that bibliometrics and cost-benefit analysis provide, but even within those domains many technical problems remain. With a view to societal impact, many RI managers lack a clear concept or a mission that is sufficiently articulated to attach valid measurements or indicators. Accordingly, most of them remain unaware and do not consciously collect information on the different types of exchanges by means of which their research infrastructure interacts to society.
Third, there is surprising agreement on at least some fundamental indicators that many RI managers would like to see collected. At the same time, their motivation for this is split between information that they would like to see collected with a view to their mission and such that they agree needs to be collected for policy makers - without necessarily making sense in substance. Overall, there is a remaining wariness to define too many indicators that are essentially "off mission" - in particular in a situation where broad missions allow many indicators that satisfy both policy and management needs (see above).
Fourth, it becomes increasingly clear that the logics of impact creation and causation are fundamentally different between single-site and distributed virtual infrastructures. While they still have to contribute to the same overarching societal targets and hence impact domains, they do so through notably different impact pathways. Overall, their function as game-changers not only within the domain of research infrastructures but for research at large remains substantially underacknowledged. Currently, we have very limited means to document the new avenues of co-creation that they enable.
Fifth, there is a notable tension between (strict) principles of open access and the abovementioned need to document complex network effects. Even if legally barred from IP-tracing users, distributed and virtual RIs, have a broad array of IT-based means available to understand - or at least document - the networks that they are creating - which in turn would allow assumptions on pathways and impact created. In practice, however, they tend to forego most this great potential by deciding against any process of registering users for reasons based on (strict) open access credentials.
Finally, the workshops provided general confirmation on the array of impact pathways as well as many of the indicators that this project has so far put forward - including important elements of semantic fine-tuning in both areas. Taking into account the needed differentiations between single-sited and distributed infrastructures and certain differences between RIs in "hard" vs. "soft" science, the RI-PATHS project's conceptual model has now arrived at a degree of maturity that allows to initiate concrete piloting efforts in collaboration with RI partners.
Publications
Research infrastructures boost to industry and economy. In a paper on L'industria, RI-PATHS’ staff Silvia Vignetti and Francesco Giffoni discuss how to estimate the socioeconomic impact of Research Infrastructures and investments in science. They argue that evaluation is essential to justify publicly-funded and costly investments yet challenging. Their critical review of existing methodologies to assess the impact of RIs focuses on the Cost-benefit Analysis (CBA) as one of the analytical frameworks for evaluating welfare changes attributable to RIs. Indeed, the use of shadow prices, along with other techniques (e.g. willingness-to-pay and avoided costs), captures the social opportunity costs of goods and services and allows the evaluation of non-market impacts.
CSIL’s Massimo Florio, Francesco Giffoni, Anna Giunta and Emanuela Sirtori have recently published a study on how public procurement by big research infrastructures enhances suppliers’ performance on Industrial and Corporate Change. Using survey data on 669 CERN’s suppliers, they built a unique data set to analyse the determinants of suppliers’ sales, profits, and development activities. They found out that collaborative relations between CERN and its suppliers improved suppliers’ performance and increased positive spill-overs along the supply chain. This insight suggested that public procurement, as mission-oriented innovation policy, should promote cooperative relations and not only market mechanisms.
For another research, Massimo Florio, Francesco Giffoni and Jessica Catalano interviewed 1,022 undergraduates to study the drivers of preferences for paying for basic research, which are still little known. Moreover, taxpayers are usually the ultimate funders of large-scale research infrastructures, but the expected discoveries of such projects often do not have any known use-value. CSIL’s researchers focused on the LHC at CERN, where the Higgs boson was discovered and found out that income, awareness, and positive attitudes towards science drive a positive willingness-to-pay for science. Social sciences and the humanities’ students resulted willing to contribute to fundamental research at least as much as their peers in science curricula. Findings suggest support to government funding of basic research as a public good. Their article "Should governments fund basic science? Evidence from a willingness-to-pay experiment in five universities” was published by Taylor & Francis Group's Journal of Economic Policy Reform.
RI-PATHS at the AESIS Impact of Science Conference
On 6th June 2019, Henning Kroll participated in the AESIS Conference in Berlin as an invited speaker, presenting findings from the RI-PATHS projects in a session on co-creation. In his presentation, he emphasised various forms of underacknowledged interactions along impact pathways. The subsequent discussion reaffirmed earlier findings in the project such as the transformative nature of distributed infrastructures and the potential tension between open access and impact documentation. In the larger debate on the conference, the RI PATHS contributions positioned itself clearly in favour of studying pathways and understanding logics - against big data inspired proposition that understanding the concrete mechanisms of causation may in fact no longer be necessary as long as outcomes can be predicted. Nonetheless, the conference demonstrated that big data approaches may provide relevant inspiration for subsequent project work with a view to tracing impact and impact pathways in area previously unaccessible.
The Network for Advancing and Evaluating the Societal Impact of Science (AESIS) is an international, open community for various types of professionals working on stimulating and demonstrating the impact of science on economy, culture and well-being. It seeks to contribute towards an increased sharing of best practices to develop effective instruments for evaluating and advancing the societal impact of science. The Impact of Science conference 2019 took place from 5th to 7th June in Berlin, Germany, involving high-level participation from the German Chancellery, the German Federal Ministry of Education and Research, several Research Councils, leading Research Organisations and diverse private sector analysts (http://aesisnet.com/event/ios19/).
Engagement with other related projects
In the course of the project, members of the RI-PATHS team have engaged with similar EU-funded projects to source further inspiration on the concept and methodologies for impact assessment. In particular, there were exchanges with the projects ACCELERATE (www.accelerate2020.eu, led by CERIC-ERIC), ResInfra (http://www.interreg-danube.eu/approved-projects/resinfra-dr, led by Centre for Social Innovation) and Data4Impact (www.data4impact.eu, led by PPMI) at both working and strategic levels.
Calendar of events
Past events where RI-PATHS project was presented
- ESFRI RIs-EOSC Liaison Workshop (30 January 2019, London, UK) Learn more
- ESFRI as the Strategic Forum to Develop a Common Methodology for Monitoring Research Infrastructures (31 January 2019, London, UK) Learn more
- RI-PATHS Participatory Workshop at DESY (8 May 2019, Hamburg, Germany) Learn more
- RI-PATHS Participatory Workshop at ELIXIR (9 May 2019, Cambridge, UK) Learn more
- RI-PATHS Participatory Workshop at ALBA (10 May 2019, Barcelona, Spain) Learn more
- Impact of Science 2019: Understanding causalities, correlations and pre-conditions for the different dimensions of societal impact of science (5-7 June 2019, Berlin, Germany) Learn more
- RI-PATHS Participatory Workshop at MERIL-2 Conference (25 June 2019, Lisbon, Portugal) Learn more
- International Future Circular Collider (FCC) Conference (24-28 June 2019, Brussels, Belgium) Learn more
UPCOMING EVENTS WITH RI-PATHS PARTICIPATION
- ESFRI VALIDATION WORKSHOP ON "MONITORING OF RESEARCH INFRASTRUCTURES METHODOLOGY AND KEY PERFORMANCE INDICATORS” (3 July 2019, Brussels, Belgium) Learn more
- RI-PATHS VALIDATION WORKSHOP (22 October 2019, Brussels, Belgium) Learn more