Dear Colleagues, Welcome to this FAO e-mail conference on "Approaches and methodologies in ex post impact assessment of agricultural research: Experiences, lessons learned and perspectives" !! Thank you for joining. You can send messages now (send them to [log in to unmask]). Messages will be posted from Monday 5 May onwards while the last day for receiving messages will be Sunday 1 June 2014. We hope that the conference will be interesting, constructive and beneficial and we encourage you to participate actively. On joining the conference, subscribers received a Welcome Text which also contains the Guidelines for Sending Messages. Here, we would like to briefly remind you of some of the main points about the running of the conference: 1. Participants should introduce themselves briefly (2-3 sentences) when sending their first message to the conference. They should also provide their full work address at the end of the message. When a message is posted, we will replace @ in the e-mail address with (at) to avoid spamming. 2. Messages should not exceed 600 words 3. People posting messages are assumed to be speaking on their own behalf and not on behalf of their employers (unless they indicate otherwise) 4. Messages posted in the conference will also be made available on the web, at https://listserv.fao.org/cgi-bin/wa?A0=Impact-L 5. No messages will be posted with attachments. If you receive a message during the conference with an e-mail attachment, just delete it without opening the attachment. 6. The Background Document to the conference, sent by e-mail to subscribers of this conference on 30 April, sets the scene for the conference. We strongly encourage you to read it, especially Section 4 (reproduced below) which provides specific guidance about the questions that participants should address in the conference. The document is available at http://www.fao.org/fileadmin/user_upload/oek/research/pdf/Background.pdf (120 KB). Contact me (at [log in to unmask]) if you want to receive the document by e-mail. After the conference, a summary document will be published, providing an easily readable synthesis of the main discussion points and conclusions. Finally, we encourage you to tell any potentially interested colleagues or contacts about this conference. A short notice is included below for this purpose. With our sincere best wishes for a successful conference, John John Ruane, PhD Research and Extension Unit, UN Food and Agriculture Organization (FAO), Viale delle Terme di Caracalla, 00153 Rome, Italy E-mail address: [log in to unmask] Web: http://www.fao.org/nr/research-extension-systems/en/ IMPRESA: http://www.impresa-project.eu/ ***************** Impact assessment of agricultural research - An FAO e-mail conference The UN Food and Agriculture Organization (FAO) is one of nine partners participating in a project on the Impact of Research on EU Agriculture (IMPRESA), funded by the European Union (EU) Seventh Framework Programme. In this recently-launched project, FAO is leading work package 1 (on Concept Development and Learning), which has as one of its main aims the establishment of a common framework to update concepts and methodologies for impact assessment of agricultural research. As part of the work package's activities, FAO will host a moderated e-mail conference about impact assessment of agricultural research from 5 May to 1 June 2014. Evidence suggests that investments in agricultural research play a key role in raising agricultural productivity and enhancing sustainability. In recent years, ever-increasing importance has been given to assessing the impact of agricultural research. This is for reasons of accountability and transparency as well as to ensure that limited financial resources are used in the most effective manner possible. Impact assessment can, in addition, provide empirical evidence of the effectiveness of past investments and thereby ensure the continued support of governments and donors to agricultural research in the future. This e-mail conference will allow participants from around the world to share and discuss their experiences, lessons learned and perspectives regarding impact assessment of agricultural research (note, the term 'agricultural research' encompasses research in the crop, livestock, forestry, fishery and aquaculture sectors). The kinds of topics to be discussed will include use of different approaches, models and tools to assess the impacts of agricultural research and best practices for disseminating impact assessment results to policy-makers and other relevant stakeholders. The e-mail conference will focus on ex post (looking at the impacts after the agricultural research is completed) and not on ex ante impact assessment (predicting the impacts that agricultural research might have in the future). The conference title is "Approaches and methodologies in ex post impact assessment of agricultural research: Experiences, lessons learned and perspectives". The conference is open to everyone, is free and will be moderated. To subscribe to the conference, send an e-mail to [log in to unmask] with the following one line in the body of the message (leave the subject line blank): subscribe impact-L firstname lastname Where firstname and lastname refer to the person's first and last name. For example, if the subscriber's name is John Smith, then the line should be: subscribe impact-L John Smith The background document to the conference was published on 30 April and can be downloaded from http://www.fao.org/fileadmin/user_upload/oek/research/pdf/Background.pdf (125 KB). It aims to provide an easily-understandable introduction and brief overview to the conference topic that participants will find useful for the e-mail conference. After the conference, a summary document will be published, providing an easily readable synthesis of the main discussion points and conclusions. For more information, please contact [log in to unmask] ******************** [FROM THE BACKGROUND DOCUMENT] 4. Main questions to be discussed by participants in the conference This e-mail conference enables participants from around the world to share and discuss their experiences, lessons learned and perspectives regarding the concepts, approaches and methodologies for ex post impact assessment of agricultural research. Note that the term 'agricultural research' encompasses research in the crop, livestock, forestry, fishery and aquaculture sectors, so the conference covers epIA relating to any of these sectors. Note also that the aim of the conference is not to discuss the potential positive or negative impacts of any specific technologies or products derived from agricultural research. Instead, the conference focuses on the concepts, approaches and methodologies used for epIA of agricultural research. The main kinds of topics to be discussed by participants in the conference are described below: 4.1 Approaches and methodologies for epIA As seen in Section 2, for epIA of agricultural research, both at the macro- and micro-level, a range of approaches and methodologies are available, which have different statistical properties, data requirements and practical characteristics. - What have been participants' experiences and lessons learned from applying different methodologies (including general issues such as what have been the difficulties in using them?; how was the complexity of the processes from inputs to impacts dealt with?; what methods were used to identify the beneficiaries of the impacts?; how were reliable and good-quality indicators identified?) - Also, for the future, what perspectives do the participants see for the different methodologies? 4.2 Assessment of non-economic impacts As seen in Sections 2.1.3 and 2.2.5, for both macro- and micro-level epIA respectively, there is increasing interest in assessing the impacts of agricultural research on environmental (including natural resources management and sustainability), social (including poverty and hunger alleviation), government policy and other non-economic dimensions, although these assessments can represent considerable data and methodological challenges. - What cost- and time-effective approaches and methodologies can be used to best address the challenges of assessing non-economic impacts? - What perspectives are there that epIA studies of non-economic impacts will become easier and more commonplace in the future? 4.3 Quantitative versus qualitative methods As described in Section 2.2, quantitative and qualitative methods have different strengths and weaknesses and their deployment involves use of different skills by the people carrying out the epIA. The importance of their relative roles in impact assessment is also a topic of considerable debate. - For epIA of agricultural research, what are the relative merits of the two kinds of methods? - When are qualitative methods to be preferred? - When are mixed methods, combining quantitative and qualitative, to be preferred (considering also the issue of cost of carrying out the assessment)? - Are there quantitative or qualitative methods currently used for epIA in other areas which might be particularly useful for epIA of agricultural research? 4.4 Randomised controlled trials (RCTs) versus other methods (quantitative or qualitative) As described in Section 2.2.1, the drive for more evidence-based and rigorous impact assessment, particularly by donors, has meant that use of RCTs has been championed in a wide range of areas and their potential role is also of considerable interest for epIA of agricultural research. - How important are RCTs likely to be for epIA of agricultural research and when are they likely to be particularly advantageous? - How important, cost- and time-effective are epIA studies from RCTs compared to those from other quantitative or qualitative methods, including qualitative narrative stories? 4.5 EpIA in the different food and agricultural sectors As seen from Alston et al. (2000), the overwhelming focus of epIA studies in the past has been on field crops, with about 1,000 estimates of economic rates of return found in the literature compared to over 300 for livestock and less than 100 for research with a natural resources focus, including forestry and fisheries. According to Raitzer and Kelley (2008), the uneven coverage of epIA work across the different food and agricultural sectors might be one of the reasons why epIA results are not crucial for donors when deciding on research funding allocations. - Is the uneven focus of epIA work across the crop, livestock, forestry, fishery and agro-industry sectors an important issue? - If so, what incentives can be provided to ensure that all the food and agricultural sectors are better covered? - To what degree can epIA methods developed for one sector be transferred to others? 4.6 Cost-effective epIA As described in the Introduction, organizations carrying out agricultural research are under increased pressure to assess the impact of their research activities. There is also a drive to measure indicators that are not purely of an economic nature and to carry out the epIA work in a more comprehensive and rigorous fashion, which may require gathering data on several relevant indicators and using more than one method (e.g. combining qualitative and quantitative methods). This kind of work entails use of financial and human resources which might otherwise be used for other purposes, such as carrying out research. For example, in their book with case studies of the impact of agricultural research on poverty, Adato and Meinzen-Dick (2007) indicate that each case study costed around 200,000 USD. While noting that the sum is relatively small compared to the total research budget, they say "the challenge now is to evolve and adopt cost- and time-effective approaches and methods...". - How can meaningful and high-quality epIA studies be carried out which do not cost too much money? - What is the appropriate proportion of a research budget to spend on epIA? - With very limited research funding, is it worthwhile dedicating resources to epIA? - What is the minimum budget one needs to carry out a meaningful epIA of agricultural research? - Are there good examples of cost- and time-effective approaches and methods for epIA of agricultural research? 4.7 Communicating the epIA findings As seen in Section 3, limited information seems to be available about how the final results of epIA studies are communicated to donors or policy-makers and how they then use these results. - What have been participants' experiences and lessons learned from communicating the epIA findings to donors or policy-makers? - What is the best approach and format for presenting the findings to them? - How can it be ensured that the results will be read and considered by donors and policy-makers? - What examples are available which demonstrate that the results of epAI studies have been used in a policy process or to make strategic funding decisions? 4.8 EpIA as one component in the overall evaluation package The primary purpose and role of epIA of agricultural research has been critically examined on occasions (e.g. Ekboir, 2003) and continues to be a topic of current debate. - What is the primary purpose of epIA of agricultural research today? - Compared to the other components in the overall evaluation package, how important is epIA today and how important is it likely to be in the future? ######################################################################## To unsubscribe from the Impact-L list, click the following link: https://listserv.fao.org/cgi-bin/wa?SUBED1=Impact-L&A=1