Impact Evaluation: Assessing the Impact of Policy-Oriented Social Science Research

The International Food Policy Research Institute (IFPRI) - a USA-based institute carrying out research, capacity-strengthening, and policy communication for food security and poverty reduction - has worked to develop methodologies for assessing the impact of social science research. Two symposia - one conducted in 1997 and one in 2001 - brought researchers together to examine how to measure the economic impact of policy-oriented social science research, and how to enhance the effectiveness of such research in policymaking environments.

A 4-page summary brief emerged from those gatherings; it offers guidance for evaluators, including:


Measuring Impact:

• While it is becomes more difficult to attribute policy responses to individual actors or pieces of research as one moves "up" various levels - from a project, to a programme, to an institution, to an entire body of knowledge - case studies that look at larger scopes and longer terms are key features of strategic policy research.
• "Anticipatory research that alerts policymakers to possible future scenarios and surprises also can be extremely valuable in reducing the time lag before
  welfare-enhancing policy changes are made."
• When choosing indicators, "Generally the consensus is that portrayal of distributional outcomes in policy-oriented social science research is more influential in effecting policy changes than portrayal of deadweight economic losses from current or alternative policies."
• Case studies raise questions about such strategies as random sampling or purposive sampling ("cherry-picking") in selecting projects and programmes to evaluate. "Each
  approach has its pros and cons, and no clear consensus seems to have emerged."
• "Measuring the impact of research requires attention to assessments both before policy change (ex ante) and after (ex post)."

Enhancing Impact:

• A communications strategy is needed from the outset so that social scientists can convey their message effectively to the particular audience(s) being addressed. "Advocacy is a powerful influence, but researchers must walk a fine line between advocating specific policies and retaining credibility as objective analysts."
• Work with political scientists and sociologists to develop an understanding of the policy process.
• "To achieve sustainable impacts from policy-oriented social science research in developing countries, strengthening the capacity of national institutions is vital..."
• "The most influential policy-oriented social science research appears to be research that presents the distributional consequences of alternative policy options....Another valuable output is primary data, especially at the household level, together with simple statistical analyses that set the stage for later, more sophisticated research, including modeling."
• "Policymakers seem to respond better to research that emanates from institutions that have a reputation for quality, credibility, and objectivity."
• "Correctly anticipating the major policy issues of the future is one of the primary ingredients in establishing priorities for policy-oriented social science research."

Editor's Note: A more detailed report on the workshop described here is available as "Synthesis Report of Workshop on Assessing the Impact of Policy-Oriented Social Science Research", by James G. Ryan; click here to access this 15-page report in PDF format.