Dear Colleagues,
This conference began on Monday 5 May while the last day for receiving messages will be Sunday 1 June 2014. We have therefore just passed the half-way stage in this 4-week conference on "Approaches and methodologies in ex post impact assessment (epIA) of agricultural research: Experiences, lessons learned and perspectives".
At this point in time, I would like to take this opportunity to thank all of you who have already sent in messages and who have dedicated your time and effort to sharing your knowledge and experience with the over 600 people who have subscribed themselves to this conference. From the contacts I have had with many of them, I know that these contributions, coming from people living in 27 different countries, have been much appreciated. I hope that you will continue participating actively in this second half of the conference.
I also hope that those who have not already done so, will contribute to the conference. The main questions to be addressed by participants in the conference are provided in Section 4 of the conference background document (reproduced below). Some of these questions, particularly about cost-effective epIA (4.6); epIA in the different food and agricultural sectors (4.5); and assessment of non-economic impacts (4.2) have received very little attention so far.
In addition, some of the messages that were posted requested specific feedback on epIA-related issues and your specific responses to these would also be appreciated. These include:
- Message 13, about quantitative methods, such as propensity score matching, to assess the impact of projects.
- Message 19, requesting feedback about the appropriate productivity measures to use for macro-level assessments of agricultural research
- Message 20, about evaluation studies which specifically searched for disconfirming evidence to be built into selection of comparative cases
- Message 31, about 4 basic issues regarding macro-level epIA of agricultural research (i.e. source of research funding; different kinds of research and thus different time lags; objectives of the research funding; and counterfactuals at the macro-level)
- Message 40, about counterfactuals with economic surplus and similar methods
- Message 42, about specific epIA protocols for seed and planting materials
I remind you that all of the 54 messages posted so far are available on the web, at https://listserv.fao.org/cgi-bin/wa?A0=Impact-L and that, after the conference, a summary document will be published, providing an easily readable synthesis of the main discussion points and conclusions. The document will be made available on the FAO website when finalised.
With all best regards
John
*********************************
[FROM THE BACKGROUND DOCUMENT, http://www.fao.org/docrep/019/as549e/as549e.pdf]
4. Main questions to be discussed by participants in the conference
This e-mail conference enables participants from around the world to share and discuss their experiences, lessons learned and perspectives regarding the concepts, approaches and methodologies for ex post impact assessment of agricultural research. Note that the term 'agricultural research' encompasses research in the crop, livestock, forestry, fishery and aquaculture sectors, so the conference covers epIA relating to any of these sectors. Note also that the aim of the conference is not to discuss the potential positive or negative impacts of any specific technologies or products derived from agricultural research. Instead, the conference focuses on the concepts, approaches and methodologies used for epIA of agricultural research.
The main kinds of topics to be discussed by participants in the conference are described below:
4.1 Approaches and methodologies for epIA
As seen in Section 2, for epIA of agricultural research, both at the macro- and micro-level, a range of approaches and methodologies are available, which have different statistical properties, data requirements and practical characteristics.
- What have been participants' experiences and lessons learned from applying different methodologies (including general issues such as what have been the difficulties in using them?; how was the complexity of the processes from inputs to impacts dealt with?; what methods were used to identify the beneficiaries of the impacts?; how were reliable and good-quality indicators identified?)
- Also, for the future, what perspectives do the participants see for the different methodologies?
4.2 Assessment of non-economic impacts
As seen in Sections 2.1.3 and 2.2.5, for both macro- and micro-level epIA respectively, there is increasing interest in assessing the impacts of agricultural research on environmental (including natural resources management and sustainability), social (including poverty and hunger alleviation), government policy and other non-economic dimensions, although these assessments can represent considerable data and methodological challenges.
- What cost- and time-effective approaches and methodologies can be used to best address the challenges of assessing non-economic impacts?
- What perspectives are there that epIA studies of non-economic impacts will become easier and more commonplace in the future?
4.3 Quantitative versus qualitative methods
As described in Section 2.2, quantitative and qualitative methods have different strengths and weaknesses and their deployment involves use of different skills by the people carrying out the epIA. The importance of their relative roles in impact assessment is also a topic of considerable debate.
- For epIA of agricultural research, what are the relative merits of the two kinds of methods?
- When are qualitative methods to be preferred?
- When are mixed methods, combining quantitative and qualitative, to be preferred (considering also the issue of cost of carrying out the assessment)?
- Are there quantitative or qualitative methods currently used for epIA in other areas which might be particularly useful for epIA of agricultural research?
4.4 Randomised controlled trials (RCTs) versus other methods (quantitative or qualitative)
As described in Section 2.2.1, the drive for more evidence-based and rigorous impact assessment, particularly by donors, has meant that use of RCTs has been championed in a wide range of areas and their potential role is also of considerable interest for epIA of agricultural research.
- How important are RCTs likely to be for epIA of agricultural research and when are they likely to be particularly advantageous?
- How important, cost- and time-effective are epIA studies from RCTs compared to those from other quantitative or qualitative methods, including qualitative narrative stories?
4.5 EpIA in the different food and agricultural sectors
As seen from Alston et al. (2000), the overwhelming focus of epIA studies in the past has been on field crops, with about 1,000 estimates of economic rates of return found in the literature compared to over 300 for livestock and less than 100 for research with a natural resources focus, including forestry and fisheries. According to Raitzer and Kelley (2008), the uneven coverage of epIA work across the different food and agricultural sectors might be one of the reasons why epIA results are not crucial for donors when deciding on research funding allocations.
- Is the uneven focus of epIA work across the crop, livestock, forestry, fishery and agro-industry sectors an important issue?
- If so, what incentives can be provided to ensure that all the food and agricultural sectors are better covered?
- To what degree can epIA methods developed for one sector be transferred to others?
4.6 Cost-effective epIA
As described in the Introduction, organizations carrying out agricultural research are under increased pressure to assess the impact of their research activities. There is also a drive to measure indicators that are not purely of an economic nature and to carry out the epIA work in a more comprehensive and rigorous fashion, which may require gathering data on several relevant indicators and using more than one method (e.g. combining qualitative and quantitative methods). This kind of work entails use of financial and human resources which might otherwise be used for other purposes, such as carrying out research. For example, in their book with case studies of the impact of agricultural research on poverty, Adato and Meinzen-Dick (2007) indicate that each case study costed around 200,000 USD. While noting that the sum is relatively small compared to the total research budget, they say "the challenge now is to evolve and adopt cost- and time-effective approaches and methods...".
- How can meaningful and high-quality epIA studies be carried out which do not cost too much money?
- What is the appropriate proportion of a research budget to spend on epIA?
- With very limited research funding, is it worthwhile dedicating resources to epIA?
- What is the minimum budget one needs to carry out a meaningful epIA of agricultural research?
- Are there good examples of cost- and time-effective approaches and methods for epIA of agricultural research?
4.7 Communicating the epIA findings
As seen in Section 3, limited information seems to be available about how the final results of epIA studies are communicated to donors or policy-makers and how they then use these results.
- What have been participants' experiences and lessons learned from communicating the epIA findings to donors or policy-makers?
- What is the best approach and format for presenting the findings to them?
- How can it be ensured that the results will be read and considered by donors and policy-makers?
- What examples are available which demonstrate that the results of epAI studies have been used in a policy process or to make strategic funding decisions?
4.8 EpIA as one component in the overall evaluation package
The primary purpose and role of epIA of agricultural research has been critically examined on occasions (e.g. Ekboir, 2003) and continues to be a topic of current debate.
- What is the primary purpose of epIA of agricultural research today?
- Compared to the other components in the overall evaluation package, how important is epIA today and how important is it likely to be in the future?
########################################################################
To unsubscribe from the Impact-L list, click the following link:
https://listserv.fao.org/cgi-bin/wa?SUBED1=Impact-L&A=1
|