Loading

UNHCR Evaluation Service Annual Report 2020 evaluating what works for the people we serve

2020 in review

  • Completed 10 evaluations: 5 centralized, 4 decentralized and 1 inter-agency
  • Initiated 13 new centralized evaluations, each one aligned with UNHCR’s priority areas
  • Developed guidance for decentralized evaluations, to be rolled out in support of the new Results-based Management system in 2021
  • Established a regional presence by out-posting 3 Regional Senior Evaluations Officers to the Americas, East and Horn of Africa and Great Lakes region, and the West and Central Africa region
  • Our team has grown from 7 staff in 2018 to 12 in 2020.

Introduction

2020 was a productive year for UNHCR’s Evaluation Service. The year began with a commitment to improve further our methods and products thereby increasing the relevance and value of our evaluative work.

When COVID-19 struck, we had to adapt quickly to respond to the new challenge and support UNHCR in responding to the pandemic.

Despite travel restrictions and other impediments, we were able to increase our agility and flexibility and provide quick support to UNHCR by identifying lessons from earlier health emergencies, such as the response to Ebola.

In this year-end report, we share with you the progress we made towards achieving our strategic goals as set out in the 2018-2022 Evaluation Strategy:

  1. Increasing evaluation coverage and quality
  2. Improving the relevance and utilization of evaluation findings
  3. Building capacity to undertake and use evaluations
  4. Strengthening an evaluation function that is linked to other complementary functions, such as oversight, strategic planning, monitoring, data and analytics, and results-based management.

1.Evaluation Coverage and Quality

Back in 2016, when the Evaluation Strategy was being developed the Service produced four evaluations. Since then, the number and coverage of evaluations each year have been increasing, with ten evaluations completed in 2020. The Evaluation Service’s projected expenditure for 2020 was $5,8 million.

There are two types of UNHCR evaluations:

  • Centralized evaluations are conducted centrally by the Evaluation Service and managed by staff at headquarters in Geneva and Senior Evaluation Officers based in regional bureaus
  • Decentralized evaluations are those managed outside the Evaluation Service by regional, multi-country and country operations and headquarter divisions. Evaluation Service provides support and quality assurance to the process.

In 2020, centralized evaluations continued to look at thematic issues such as organization-wide policy or thematic areas of strategic significance to UNHCR, major UNHCR emergency response (L3) and country operations.

Decentralized evaluations looked at UNHCR’s response on programmatic issues, focusing on a particular division, region, country, or a specific programme or project.

We also provided tailored support for exercises that, while not evaluations in themselves, were evaluative in nature and required objective assessment approaches. Examples included the desk review UNHCR’s leadership and coordination in refugee response settings and the Independent Donor Assessment Review. This latter was undertaken with the Division of External Relations.

In 2020, the Evaluation Service mapped the geographic coverage of evaluations conducted over the past five years. This showed that almost half of UNHCR’s operations have been evaluated during the period (over 60 countries have been covered by evaluations and evaluation case studies over the period). The deployment of Regional Senior Evaluation Officers will further strengthen the coverage and relevance of evaluations conducted in the field.

2. Improving relevance and utilization of evaluation findings

Utility of evaluations is key to our work. Without uptake and use, even the most high-quality evaluation may be seen as a waste of scarce organizational resources.

The essence of our work is to undertake strategic analysis and to provide robust evidence that will inform management decision-making to the benefit of the organization and, ultimately, to Persons of Concern.

Here’s a short video, in which UNHCR’s Deputy High-Commissioner, Kelly Clements talks about the impact our evaluations had on the organization.

Findings from the selected evaluations

UNHCR’s Efforts to Promote Refugee Inclusion and Engagement with Development Actors

The Evaluation Service is currently evaluating how UNHCR engages in humanitarian development cooperation.

Cooperation with development organizations has been part of UNHCR’s approach to building long-term support and finding durable solutions for people forced to flee. Over the past few years, UNHCR has deepened its cooperation with global entities like the World Bank, regional entities like the African Development Bank, UN development agencies and development donors.

Here are some of the emerging findings from the evaluation:

1) Further expand engagement with development actors

UNHCR is already engaging systematically with development actors in many areas, but there are opportunities to engage further, such as with the UN Country Team and the reformed Resident Coordinator function. We could also engage with peacebuilding and conflict prevention actors to strengthen the protection components in their initiatives.

2) Close the gaps in operational practice

There is broad buy-in across the organization to pursue more comprehensive responses to humanitarian development cooperation. UNHCR has made institutional investments in this area by, for example, introducing Senior Development Officer positions. However, UNHCR could further strengthen this agenda more broadly across the organization and consider having a more consistent stance on how to approach financial relations with development actors.

3) Strengthen engagement with partners on protection issues

UNHCR already engages extensively on protection-related issues with development actors. We provide advice, training, data, and conduct joint advocacy campaigns. We could go further and broaden UNHCR’s engagement in this field by strengthening both UNHCR’s ongoing advisory functions to the World Bank’s funding instruments on forced displacement (IDA 18 and 19) and UNHCR’s capacity to conduct protection analysis.

4) Look at planning and budgeting processes

UNHCR’s budgetary and planning systems are sound but the organization faces challenges when undertaking multi-year planning or when receiving funds from development donors. Potentially, there are different options that could be considered in line with the current Results-Based Management Renewal Project. These could include further decentralising spending authority for development resources; ensuring adequate funding within the Operating Level for core facilitation activities on development cooperation and; enabling multi-year partnership agreements.

UNHCR's Approaches to Workforce & Partner Learning and Development

The evaluation explored the performance of UNHCR’s overall organizational ‘learning system’ and its capacity to adapt over time to the changing context of UNHCR.

The evaluation found that UNHCR’s learning service was one of the most advanced among other UN agencies. But there are opportunities to further improve the alignment with regionalization, and to support an environment where learning is embedded in the day-to-day flow-of-work, and linked more clearly to organizational performance.

The evaluation highlighted UNHCR’s commitment to equality of learning. This is highly valued by staff, but there are practical impediments to it being realized. These include:

  • Learning offers in languages other than English are limited
  • Inequalities of access especially at lower grades and local positions
  • Uneven management support for learning across the organization
  • Literature suggests people learn from peers not specialists - examples of staff developing innovative practices to do this were documented by the evaluation but they fall outside of the Global Learning and Development Center (GLDC) learning offer and formal strategy
  • Utilization rates are low for much of online learning on offer, with 42 per cent of those enrolling in a programme not actually completing it – often for work-related reasons.

The evaluation calls for a transformation in knowledge development and skill acquisition so that:

  • Learning provision is more nimble and more responsive to the learning needs in emergency operations and new context and needs
  • Learning is more firmly embedded and monitored against UNHCR’s strategic priorities, annual planning and budgeting processes
  • Responsibility and resources for learning is shared more widely beyond GLDC with learning and development capability also being developed in the Regional Bureaus
  • The current emphasis on training and formal learning shifts to greater support for ‘learning-in-the-flow-of-work’, with a greater focus on bite-sized/micro-learning and more mentoring, coaching and Communities of Practice
  • GLDC’s role evolves to become less focussed on formal training provision and more focussed on supporting learning at the regional and country level, advising Bureaux and operations, and providing quality assurance
  • Learning is more central to UNHCR’s recruitment, career planning and performance appraisal processes
  • Managers are required to support the learning and development of their supervisees. This is integral to all management and leadership programmes.

UNHCR’s Response to the Venezuela Refugee Crisis

The evaluation of the Venezuela situation explored the extent to which UNHCR provided a timely and effective response to the needs of refugees and migrants and examined what enabling and constraining factors existed.

The evaluation found that UNHCR’s assistance and protection strategies were highly relevant and effective. While conditions and achievements vary across Colombia, Ecuador, Brazil and Peru, findings showed that the focus on support to the most vulnerable; the provision of assistance through helplines and cash-based interventions; and strengthening of asylum systems and mobilization of support networks, produced a relevant and effective response.

Lessons and recommendations:

  • Bolster communication and community-based protection at irregular border entries
  • Identify more good practices in virtual service delivery under COVID-19, such as telephone hotlines, chat apps and virtual CBI delivery
  • Explore strategies to maintain and institutionalize capacity-building with governments and partners despite high staff rotation in these agencies
  • Increase monitoring of partner capacity and detect potential rights violations in service delivery

The evaluation highlighted that UNHCR’s delivery of mid-to-long term solutions was nascent but uneven.

The findings revealed that UNHCR’s approach to socio-economic inclusion was limited. Likewise, there was a general lack of clarity on how programmes targeting persons of concern could transition from humanitarian assistance to longer-term development initiatives and what the effectiveness and impact had been of livelihoods and other solutions-based activities which were implemented.

Lessons and recommendations:

  • Strengthen solution-based activities in cities further away from borders. These areas may present better opportunities to implement forward-looking protection and solutions initiatives
  • Increase resource allocation and efforts in partnership development towards the region’s socio-economic inclusion strategy
  • Leverage UNHCR’s ongoing advocacy efforts to strengthen regularization and inclusion into local and national development systems
  • Expand evidence base on good practice for socio-economic inclusion and other solutions approaches.

The evaluation also finds that co-leadership of the response, data-sharing and communication externally and internally remain challenging.

Lessons and recommendations:

  • Document and communicate UNHCR’s roles as catalyst, implementer and communicator and share with stakeholders
  • Evaluate the inter-agency platform and response to provide lessons for future action and for other joint responses
  • Seek data-sharing agreements and mechanisms with governments and partners to address data gaps and overlaps
  • Facilitate opportunities for regular cross-country exchanges of experiences, ideas and good practices.

UNHCR - Government of Colombia Four-Year IDP Plan

In 2020, UNHCR in Columbia commissioned a decentralized evaluation of the UNHCR - Government of Colombia Four-Year Internally Displaced Persons (IDPs) Plan (2015-2019).

In this interview Jozef Merkx, UNHCR Representative in Colombia shares his experience with the evaluation

COVID-19

We are actively studying UNHCR´s response to COVID-19. Within the first three months of the pandemic, looking at previous UNHCR responses as well as the evaluative work of others we collated evidence, some of which fed into COVID-19 Response: Lessons from UNHCR’s Evaluation Evidence.

We have also distilled lessons from UNHCR’s Ebola responses in refugee settings. The lessons predicted increased mental health problems and sexual and gender-based violence risks. They also emphasized the importance of good data and strong coordination mechanisms and the need for significant recovery support for livelihoods and local economies. These findings helped UNHCR adapt current operations to the COVID-19 context.

Going forward, we will be leading or supporting three COVID-19 related evaluations in 2021:

  • a meta-evaluation on UNHCR’s adaptation and response;
  • a joint-evaluation on the protection of refugee rights during COVID-19;
  • as a member of the inter-agency humanitarian evaluation of the Global Humanitarian Response Plan.

3. Building capacity to undertake and use evaluations

Support to decentralization and regionalization

To support the decentralization and regionalization process currently underway at UNHCR, in 2020 we extended our regional presence by out-posting two new Regional Senior Evaluations Officers to East and Horn of Africa and Great Lakes region; and to West and Central Africa.

We plan to expand coverage to all regions by 2023.

Regional Senior Evaluation Officers work closely with regional bureaus on developing and implementing regional evaluation plans; providing technical advice, quality support and coaching on evaluation across regions.

Management responses

With respect to follow-up on recommendations, there is scope for improving the timeliness of management responses to evaluations. While the evaluation policy requires submission within two months, responses are being submitted 120 days after an evaluation on average. We are looking at ways to speed up the action planning process and and have started monitoring evaluation use and uptake one year later.

Capacity building

We are also working on building evaluation skills capacity across UNHCR. In 2020, the Evaluation Service in partnership with the Global Learning and Development Centre organized a webinar. This focused on decentralized evaluations, their purpose, how to conduct them, what stages each evaluation includes and more. Over 140 colleagues from the field working in country offices and regional bureaus participated in the webinar. We are looking at other opportunities to further professionalize evaluation among mid-level personnel as we move forward - including within the Results-Based Management policy roll-out and training process.

Communication and outreach

We have also invested in strengthening our outreach and communications activities. We have developed our first communications strategy, with the ambition of strengthening engagement with different audiences. Going forward, we will be focusing our work on preparing more targeted evaluation products and utilizing different tools to reach out to stakeholders across the organization and beyond.

4. Strengthening Linkages

As is foreseen in the Evaluation Strategy, we are seeking stronger linkages with other functions, such as oversight, strategic planning, monitoring, data and analytics, and results-based management as well as with the wider UN system.

Strategic planning and Results-Based Management (RBM)

UNHCR´s new strategic planning and RBM policy is being rolled out in 2021. Country Strategy Evaluations will be an important tool enabling Country Operations to take stock of achievements, distil valuable lessons, and hone in on priorities in the lead up to new Multi-Year Strategic Planning process (MYSP).

Oversight

At the end-2019 a new policy on independent oversight in UNHCR was approved. It aimed to promote and enhance organizational accountability – while recognizing the different roles that the evaluation, audit, inspection, and other functions play in promoting learning and ensuring accountability. Over 2020, the Heads of oversight functions – including the Head of the Evaluation Service - met regularly to synchronize oversight activities and coordinate the oversight response to the COVID-19 pandemic.

Linkages with the wider UN

UNHCR continues to contribute in a systematic way to joint UN evaluation efforts, playing a strong technical and leadership role.

Last year, UNHCR’s Head of Evaluation was selected as Vice-Chair of the Executive Steering Committee in United Nations Evaluation Group (UNEG).

Acting as UNEG focal point, the Head of Evaluation plays an influential role and advocates for refugees and other displaced groups to be central in planned evaluations.

In 2020, UNHCR participated in a number of joint system-wide evaluations and studies:

We are also coordinating evaluation work on COVID-19 with the Organisation for Economic Cooperation and Development’s Development Assistance Committee (OECD-DAC), selected Governments and UN agencies as part of the Global Evaluation Coalition. One of the efforts will focus on examining the effectiveness of the response in the international protection of refugees following the onset of COVID-19.

The year Ahead

In 2021, we plan to complete the following evaluations:

  • UNHCR's Country Strategy Evaluations: Egypt, Zambia, Mexico, Uganda, Sudan and Multi-Country Operations in Baltic and Nordic Countries
  • UNHCRs L3 Cyclone Idai Response in Mozambique, Malawi and Zimbabwe
  • Evaluation of UNHCR-led Initiatives to End Statelessness
  • Humanitarian Development Cooperation Final Report (Year 3)
  • Implementation of UNHCR’s 2018 Age, Gender and Diversity Policy (Year 1)
  • UNHCR's L3 Response in the Democratic Republic of Congo
  • UNHCR's L3 Emergency Response in Burkina Faso, Niger and Mali
  • UNHCR's Repatriation Programmes and Activities 2015-2020
  • UNHCR's Asylum Capacity Development

** UNHCR’s response to COVID-19 will be considered systematically in the above evaluations and learning distilled

Joint Evaluations:

  • Joint (UNAIDS) Programme’s work on preventing and responding to gender-based violence
  • Iterative evaluation of the UNHCR/UNICEF Fair Deal (Blueprint) for Refugee Children
  • Joint Evaluation of the Protection of the Rights of Refugees during the COVID-19 Pandemic (under the auspices of the COVID-19 Global Evaluation Coalition)
  • Inter-Agency Humanitarian Evaluation L3 Yemen
  • Inter-Agency Humanitarian Evaluation of the Global Humanitarian Response Plan for COVID-19

Peer Review of UNHCR’s Evaluation Function

The High Commissioner has formally requested the Development Assistance Committee’s Evaluation Network and the United Nations Evaluation Group to conduct a professional peer review of UNHCR’s evaluation function in 2021. This will provide important elements for, inter-alia, the formulation of a new evaluation strategy 2023-2027.

Global Compact on Refugees

We recognize that Global Compact on Refugees is an important framework for the protection of the rights of refugees. We will be examining opportunities to contribute towards the high-level officials meeting on the GCR, scheduled for December 2021.

We will also seek to offer evaluation insights for upcoming important meetings which bring together the global community of decision makers.

A message from the Head of the Evaluation Service, Lori Bell

We are committed to finding new ways of sharing knowledge, learning, and evaluating what works for UNHCR and the people we serve.

Let us know how we can support your work? Contact us at hqevaser@unhcr.org