From Danielle Barret, Senior advisor to DG Research & Strategy of the “French Agricultural Research Centre for International Development” (Cirad) in charge of “Innovation-Impact Task-force” co- coordination.

I have been highly interested in the discussions launched by John Ruane’s synthetic document on impact evaluation of agricultural research. Some of us thought that the IMPRESA project was addressing the European zone. But surprisingly, most of the participants talked about impact evaluation in Southern contexts. A subject of high interest for CIRAD. [The macro-level and micro-level impact assessments that will be carried out through the Impact of Research on EU Agriculture (IMPRESA, http://www.impresa-project.eu/) project will all be from the European region. The project is divided into six work packages (WP) and FAO is leading WP1, which is looking at contemporary issues regarding agricultural research and the concepts and methodologies for impact assessment of agricultural research. Although the general objectives of the IMPRESA project are looking at agricultural research in the EU, the issues regarding concepts and methodologies to be used are general and valid for all parts of the world...Moderator].

A. What are we doing on Impact evaluation?

The ARD/AR4D (agricultural research and development/ agricultural research for development) challenge is development and since 2000, this has meant the UN Millennium Development Goals (in brief: food security; poverty alleviation; natural resources’ sustainability) soon to be turned in called Sustainable Development Goals. Working in Southern contexts and seeking for impacts means that every research should refer to these ultimate goals whenever working at global, regional, national or local, & farmer levels.

Cirad is clearly putting learning and beneficiaries & stakeholders’ demands as first objectives of impact evaluation (before accountability to donors) and is considering projects (or rather succession of projects) and programs as the subjects of evaluation (rather than the institution). Five Impact evaluation studies have been produced between 2000 and 2005 at a macro level, looking at the economic dimension of impact (Reference 1). Lack of robust data made it unsatisfactory. Four long-term projects were evaluated last 2012 (Reference 2) at both national and regional levels, looking at diverse dimensions of impacts (economic, social, organizational, sanitary). They highlighted two major points: the constitution of the productive capacity in a long run (social capital, human capital, socio-technical capital) and the intermediation role of research, to explain the contribution of Cirad to the respective outcomes and impacts of the projects. The method was mainly qualitative (Impact pathway descriptions, time-frame diagrams, storytelling).

This year, Cirad launched an “Innovation-Impact Task-force” which will enlarge the former exercise to 12 case studies (selected from around 60 projects). 

Firstly, we will thoroughly examine some methodological issues with some international experts: 
a) case studies boundaries and nested levels of observations; 
b) quanti-quali mixed methods 
c)partners’ role within impact evaluation (IE)
d) organizational/institutional impacts...

Secondly, we will discuss the two options of using a set of methods to adapt to the diversity of case studies or using a standardized method for all of them before implementing the case studies. 

Once the case studies will be finished, the aim of the Task force is to disseminate the results among partners and stakeholders and to promote an impact culture among the Cirad researchers and their southern partners. The mechanisms and milestones highlighted within the impact pathways designs and other analyses might be used for ongoing projects and ex ante for next ones having similar features. 

B. A few comments about some of the ongoing discussions in the e-mail conference

1. The subject of the evaluation: institution or project/program?

The choice is mainly linked to the objective of evaluation. Focusing on accountability to donors may explain the choice of an institution (see messages 48, 68 by M.Pareja); as learning and answering beneficiaries & stakeholders ‘demands are our first concerns, we choose to evaluate the impact of set of projects or programs. 

2. The quanti/quali and mixed method debate

Framed by J. Ruane (see Section 2.2.3 of the conference background document) this debate received interesting although controversial answers (such as messages 12, 15, 26, 41, 70, 76…) offering finally a kind of simple “recipe” (quantitative method for technical innovations and economic dimension of impact; qualitative methods for other dimensions of impact). It may sound reasonable in theory but in practice the question of data prevails. Adaptation to the evaluation topic and to the local conditions seems a basic rule.

3. Contribution vs attribution debate

(Still controversial after reading the last messages 84-86-92-95 with H. Taye, D. Suryadarma and M. Stigler's contributions). From our own practice dealing with complex and long-term case studies, attribution seems a fallacy. We have long adopted a systemic view of ARD (AIS: agricultural innovation system) which allows dynamic analyses of actors’ role in the innovation processes (Reference 3). The contribution approach can fit with this conceptual framework (see References 4 and 5).  

3- The partners’ participation to the IE  

This question received much attention (messages 5, 30, 32, 33, 51, 57, 61…especially the exchanges between A. Yapi, H-N. Nguyen, M. Stigler & D. Barjolle) of the e-conference. In my opinion, we could make a distinction between 
(i) long-term ex post impact evaluation in which local participants are either “experts” attached to the evaluation team or stakeholders & beneficiaries concerned by interviews and focus groups and 
(ii) ongoing participatory projects having designed a monitoring and evaluation (M&E) process for following up the changes arising, which are associating stakeholders and beneficiaries all along their process ( see messages 57 - S.A. Perez; 58 - M. Pareja; 63 - H. Taye; 73 - F.Zaal; 79 - L. Magingxa…)

Basically, we found from our own practices and from the literature that ex-post IE exercises are necessarily long-term ones that have no or little utility on short term research design & policy. However, they bring a lot of lessons to learn and use. So to tackle this time-lag of ARD/AR4D efficiency for beneficiaries’ sake, and having in mind time and cost constraints, the solution could be doing  less ex-post  IE but more participatory research projects equipped ex ante with concepts (such as AIS), and M&E&IA methods and tools that would ensure that outcome and impact matters and framework designs for data collection are shared with local actors from the beginning.

Finally, some of our methodological concerns have not (or little) been evoked, such as the organizational/institutional dimensions of impact or the best ways of tackling the nested dimensions of complex case studies. Other colleagues might come back on these topics.

Danielle Barret
Senior advisor to DG Research & Strategy
Co-coordinator of “Innovation-Impact task-force”
Cirad
42 rue Scheffer
75116 Paris
France
+33 (0)1 53 70 20 95
e-mail: danielle.barret (at) cirad.fr 
http://www.cirad.fr/en/home-page

References:

(1) 5 studies by Claude Freud (Cotton in Cameroon; Bananas & plantains in Cameroon; Catfish in Vietnam; SCV in Brazil)

(2) (Coffee in Nicaragua; Mango export in West Africa; Advisory services in West Africa; “Peste des petits ruminants” in Marocco) - Alami, S., D. Barret, E. Bienable and L. Temple (2013). SyntheÌse d’eìtudes de cas sur l’eìvaluation d’impact de la recherche agronomique dans les pays du sud: cirad. http://hal.cirad.fr/cirad-00904862/en  

(3) Coudel, E., H. Devautour, C. T. Soulard, G. Faure and B. Hubert, Eds. (2013). Renewing innovation systems in agriculture and food, Wageningen Academic Publishers.

(4) Mayne, J. (2012): Contribution analysis: Coming of age? Evaluation 18(3), 270-280

(5) Spaapen, J. & van Drooge, L. (2011): Introducing ’productive interactions’ in social impact assessment. Research Evaluation, 20: 211-218.
 
[To contribute to this conference, send your message to [log in to unmask] The last day for sending messages to the conference is 1 June. The searchable message archive is at https://listserv.fao.org/cgi-bin/wa?A0=Impact-L ].

########################################################################

To unsubscribe from the Impact-L list, click the following link:
https://listserv.fao.org/cgi-bin/wa?SUBED1=Impact-L&A=1