ACADEMY PUBLIC POLICIES

Summary

Measuring the Impact of Humanitarian Aid must do through a report which investigates the current state of the art in measuring and analyzing the impact of humanitarian assistance.

It is concerned with questions around how impact can be measured, why this is increasingly being demanded, and whether it is possible to do it better. It also explores the benefits, dangers and costs that paying greater attention to impact might entail.

Although questioning the impact of humanitarian assistance is not new, it has moved up the humanitarian agenda in recent years. As the overall volume of humanitarian assistance has increased, so there has been greater scrutiny of how this money is spent, while reforms within the West’s public sectors have seen the introduction of new management systems focusing on results (Macrae et al., 2002). Several UN agencies, donors, and NGOs are developing results-based management systems, and investing considerable resources in them, partly in an effort to demonstrate impact more clearly.

This increasing pressure to show results has yet to translate into clear improvements in the measurement or analysis of impact. Assessment of impact is, in fact, consistently poor.

There are, of course, many good reasons why it is difficult to measure the impact of humanitarian interventions, including difficult issues of causality and attribution and a lack of basic data, such as population figures. Relief interventions are often of short duration, capacity and resources are stretched, insecurity may limit access to populations and the space for analysis and research is constrained. Nor is the new emphasis on results without costs of its own: within the humanitarian sector, a focus on measurement could reduce operational effectiveness and lead to the neglect of issues such as protection and dignity because they are difficult to measure. Focusing on what is measurable risks reducing humanitarian aid to a technical question of delivery, rather than a principled endeavor in which the process as well as the outcome is important.

These difficulties and risks do not, however, mean that impact cannot be measured in some circumstances; where measurement in a scientific and quantifiable sense is not possible, impact can still be analyzed and discussed.

Indeed, the scientific tradition and more participatory and analytical approaches should not be seen as polar opposites, but as complementary approaches.

Definitions, objectives, and context

The question of impact within the humanitarian sector has been addressed in three main ways:

• The analysis of likely impact before the start of a project, in order to anticipate the wider consequences of an intervention.

• Ongoing analysis of impact throughout a project or as part of management systems, in an attempt to adapt interventions or monitor performance.

• Analysis of the impact of interventions after the fact, as part of evaluations or research. Impact is used as a key criterion in the evaluation of humanitarian work, and most evaluations consider it.

Impact can be analyzed at the level of individual projects, and at much broader organizational or country-wide levels. Attempts to measure impact can restrict their focus to the intended effects of interventions, or they can encompass broader indirect and unintended consequences.

There is no accepted definition of ‘impact’ within the humanitarian sector, and the definitions current within the development field, though adopted for use in the humanitarian sector, may not fully capture the particular nature of humanitarian work. In particular, the concept of change is central in developmental definitions of impact, but in humanitarian aid the aim is often to avert negative change (for example to prevent famine), rather than bring about a positive change.

This may be harder to measure.

The report, though using existing developmental definitions, uses them with due caution.

The humanitarian system’s increasing interest in impact needs to be understood in the context of broader debates about accountability in humanitarian aid, and against the background of public management reforms within Western governments.

A central element of this reform is the shift from an input–output management model towards a greater emphasis on results. Service providers not only report progress in implementing activities, but must also demonstrate that they generate some achievements. Experience from the introduction of results based or performance management systems within Western governments suggest a need for caution in adopting these approaches uncritically.

The analysis of impact should not, therefore, be seen purely as a narrow technical question about the effectiveness of individual projects; discussion about impact should not be confined to a sub-set of evaluation techniques.