Policy learning, evaluation, and aid effectiveness: Mining lessons on project and program success from the Asian Development Bank

Abstract

Lesson drawing, or learning from past policies or programs, can improve current or future policies or programs and, thereby, lead to policy success. This requires various types of evaluations that identify and highlight different causal relationships in the system. However, the literature on policy evaluation has little to say about how such evaluations work in practice. For example, in the case of overseas development assistance, although multiple studies examine factors that contribute to aid effectiveness, they do not use and build on lessons from internal evaluations of aid projects and programs. Using data on project and program evaluations from the Asian Development Bank (ADB), this paper compares the lessons from external evaluations on aid effectiveness with those of internal evaluations. It critically examines ‘lessons learned’ by the ADB in over 950 sovereign interventions across 38 countries in Asia-Pacific during 1996-2016 using relatively new ‘data science’ approach of text mining. It specifically analyzes term frequencies, proportions in evaluations of successful and unsuccessful interventions, and correlations to understand the content and content relationships of the lessons learnt. It finds that while internal evaluations validate and even go beyond several micro and meso level lessons of external evaluations – such as within country and sector variation and project characteristics of (un)successful interventions – they say less about macro level, theoretical relationships that set the context for aid effectiveness, such as per capita economic growth or the level of democracy in the borrowing country. The findings suggest the need for a multilevel evaluation framework consisting of micro, meso, and macro evaluations which pick up different factors that influence success and failure and, hence, contribute to better lesson drawing.