IMPACT-L Archives

Moderated conference on impact assessment of agricultural research: May 2014

Impact-L@LISTSERV.FAO.ORG

Options: Use Classic View

Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Topic: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Tue, 13 May 2014 10:49:30 +0200
text/plain (32 lines)
My name is James Stevenson, I'm based at FAO in the Secretariat for the CGIAR Independent Science and Partnership Council. I spend most of my time supporting the Standing Panel on Impact Assessment which has a responsibility for oversight of impact assessment activities in the Consultative Group on International Agricultural Research (CGIAR).

This is a response to Amadou's question (Message 13) about quantitative methods for assessing the impact of a project which introduces new agro-ecological farming techniques in Senegal. Amadou has exactly the right starting point for this analysis - the problem of self-selection into the project. I want to explain a bit what we mean by this and then come on to answering Amadou's question about possible solutions for it. 

What this means is that we expect the people who (when they know about the project) want to take part and then try out the farming techniques, to be different from the rest of the population of farmers. Some sub-set of the people that try out these methods will stick with them in subsequent seasons - the "adopters". Not only that, we expect adopters of the techniques to be different from non-adopters in ways that also influence the things that Amadou cares about in her analysis - incomes, crop yields and food security. This is what we mean when we talk about selection bias. A simple comparison of the incomes or crop yields for adopters and non-adopters of the practices in Amadou's project area would be biased by these differences brought about by self-selection - many other things would be different between these groups other than their choice to be an adopter. This is also referred to as an endogeneity problem.

We might expect the more skillful, more entrepreneurial or more empowered farmers to be the ones to trial and then adopt new techniques. The problem for Amadou's evaluation is that these attributes affect the outcome we care about (i.e. crop yield or income) in ways other than just via the adoption of the techniques AND that it is very hard to control for these factors ex-post in an impact evaluation. They are sometimes referred to as unobservable factors. 

Propensity score matching (PSM) is a method for trying to control the selection bias problem. Say we would like to take a sample of farmers two years after the project has finished and within that sample compare the outcomes for adopters of the techniques' Amadou is interested in to a sample of farmers who are not adopters of the techniques. PSM attempts to construct a comparison group from among the pool of non-adopters by modeling the probability of adopting on the basis of observed characteristics which are unaffected by adoption. Adopters are then matched on the basis of this probability, or propensity score, to non-adopters. A good introductory text to these quantitative methods from the World Bank is available online, by Khandker, Koolwal and Samad (2010) at https://openknowledge.worldbank.org/bitstream/handle/10986/2693/520990PUB0EPI1101Official0Use0Only1.pdf (3 MB). 

The problem with PSM is that, when we only have a single cross-sectional survey, the non-adopters in our sample have already expressed their choice not to adopt the technology so what is left over after the propensity score must explain the adoption decision - the unobservable factors are important. De Janvry, Dustan and Sadoulet (2011) wrote a very nice report on the ex-post evaluation of new agricultural technologies in which they critique the use of PSM. This report, for the Standing Panel on Impact Assessment of the CGIAR Independent Science and Partnership Council, the group I work with, is also available online, at http://gspp.berkeley.edu/assets/uploads/research/pdf/deJanvryetal2011.pdf (0.5 MB).

What de Janvry and many others in the development economics literature advocate are approaches that seek exogeneity - that is, by constructing evaluations of agricultural technologies in contexts where these unobservable factors are not liable to play such havoc with attempts to control for differences between adopters and non-adopters. Some of these methods (randomized control trials, natural experiments, instrumental variables estimation) are outlined in the World Bank introductory text I linked to earlier. This is becoming a long post, but I just wanted to get these issues out there in the group and allow others to comment. 

James Stevenson
Agricultural Research Officer
CGIAR Independent Science and Partnership Council (ISPC) Secretariat
Room C6-24, 
FAO, 
Viale delle Terme di Caracalla, 
Rome, 
Italy
+39 06 570 52251
e-mail: James.Stevenson (at) fao.org 

[To contribute to this conference, send your message to [log in to unmask] For further information, see http://www.fao.org/nr/research-extension-systems/res-home/news/detail/en/c/217706/ ].

########################################################################

To unsubscribe from the Impact-L list, click the following link:
https://listserv.fao.org/cgi-bin/wa?SUBED1=Impact-L&A=1

ATOM RSS1 RSS2