IMPACT-L Archives

Moderated conference on impact assessment of agricultural research: May 2014

Impact-L@LISTSERV.FAO.ORG

Options: Use Forum View

Use Proportional Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
Reply To:
Date:
Wed, 30 Apr 2014 16:14:10 +0200
Content-Type:
text/plain
Parts/Attachments:
text/plain (384 lines)
Dear Colleagues,

As indicated in my message to you yesterday, I am now sending you the Background Document to the FAO e-mail conference on "Approaches and methodologies in ex post impact assessment of agricultural research: Experiences, lessons learned and perspectives"

It aims to provide an easily-understandable introduction and brief overview to the conference topic that you, as participants, will find useful for the e-mail conference. The Introductory Section to the 18-page background document explains first that FAO is hosting the conference as part of its activities in the Impact of Research on EU Agriculture (IMPRESA) project. It then provides an introduction to the general area of evaluation and subsequently narrows it down to ex post impact assessment (epIA), one component of the whole evaluation package. Section 2 describes the main approaches and methodologies used for carrying out epIA of agricultural research. These are classified into two broad groups: macro-level assessments (covered in Section 2.1) looking, for example, at the impacts of agricultural research at the regional or national level and micro-level assessments (Section 2.2), looking at the impacts of specific research-derived interventions. Section 3 briefly considers the issue of how epIA results are communicated to donors and policy-makers and how they subsequently use them. Section 4 provides a series of questions which participants are asked to address during the conference. Section 5 contains references (most of which are available on the web), abbreviations and acknowledgements.

The document is provided below, and is also available on the FAO website at http://www.fao.org/fileadmin/user_upload/oek/research/pdf/Background.pdf (120 KB).

The conference begins on Monday 5 May. If any of your colleagues wish to subscribe, they should send an e-mail to [log in to unmask] with the following one line in the body of the message (leave the subject line blank):

subscribe impact-L firstname lastname

Where firstname and lastname refer to the person's first and last name. For example, if the subscriber's name is John Smith, then the line should be:
subscribe impact-L John Smith

After the conference, a summary document will be published, synthesizing the main issues discussed during the conference.

Best regards

John


John Ruane, PhD
Research and Extension Unit,
UN Food and Agriculture Organization (FAO),
Viale delle Terme di Caracalla,
00153 Rome, Italy
E-mail address: [log in to unmask]
Web: http://www.fao.org/nr/research-extension-systems/en/
IMPRESA: http://www.impresa-project.eu/

***************************************
Background document to the FAO e-mail conference on "Approaches and methodologies in ex post impact assessment of agricultural research: Experiences, lessons learned and perspectives"

John Ruane,
FAO Research and Extension Unit,
UN Food and Agriculture Organization (FAO),
Viale delle Terme di Caracalla,
00153 Rome, Italy

Abstract

This article represents the background document to the FAO moderated e-mail conference entitled "Approaches and methodologies in ex post impact assessment of agricultural research: Experiences, lessons learned and perspectives" which begins 5 May 2014. The document's aim is to provide an easily-understandable introduction and brief overview to the approaches and methodologies used in ex post impact assessment (epIA) of agricultural research. The document begins with an introduction to the general area of evaluation and then narrows down to epIA, one component of the whole evaluation package. It proceeds to describing the main approaches and methodologies used for carrying out epIA of agricultural research. These are classified into two broad groups: macro-level assessments looking, for example, at the impacts of agricultural research at the regional or national level and micro-level assessments, looking at the impacts of specific research-derived interventions. The document also briefly considers the issue of how the epIA results are communicated to donors and policy-makers and how they subsequently use them. It concludes with a series of questions which participants are asked to address during the conference. After the conference, a summary document will be published, synthesizing the main issues discussed during the conference.

1. Introduction

1.1 IMPRESA

The UN Food and Agriculture Organization (FAO) is one of nine partners in a project on the Impact of Research on EU Agriculture (IMPRESA, http://www.impresa-project.eu/) funded by the European Union (EU) Seventh Framework Programme. The project's overall aim is to measure, assess and comprehend the impact of all forms of European agricultural research on key agricultural policy goals, including farm level productivity but also environmental enhancement and the efficiency of agri-food supply chains.

The project will, inter alia,
- Survey the trends, sources and objectives of agricultural research across Europe
- Carry out in-depth analysis of research impact pathways in six case studies of innovation in agriculture
- Analyse the effects of agricultural research on productivity and other relevant economic, environmental and social dimensions in selected countries.

In this recently-launched project, FAO is leading work package 1 (on Concept Development and Learning), which has as one of its main aims the establishment of a common framework to update concepts and methodologies for impact assessment of agricultural research. As part of the work package's activities, FAO is hosting this moderated e-mail conference which begins on 5 May 2014 (https://listserv.fao.org/cgi-bin/wa?A0=Impact-L).

1.2 The growing role of evaluation

The conference focuses on impact assessment which is one component of the entire evaluation process. Evaluation has become ever-more important in recent years. According to Leeuw and Vaessen (2009): "over the last 15-20 years, governments and other (public sector) organizations have been paying much more attention to evaluation. It has become a growth industry in which systems of evaluation exist, with their methodologies, organizational infrastructures, textbooks, and professional societies".

In all domains of public sector intervention, such as agriculture, development assistance, education and human health, evaluation has become more important because of a strong push for greater transparency and towards evidence- and results-based management, where the organizations responsible for carrying out interventions are required to show that they used their resources efficiently and to demonstrate that they achieved their desired results (or can explain satisfactorily why they did not). Organizational learning may also be another important goal of evaluation.

Agricultural research is no exception to this trend. Although there are very strong arguments for substantially increasing the funding allocated to agricultural research, even at today's levels the sums involved are quite substantial (Beintema et al., 2012). In 2009, an estimated $33.7 billion, in 2005 purchasing power parity (PPP) prices, was invested in public agricultural R&D while investments from the private sector, primarily in developed countries, were also substantial and comprised an estimated 35-41% of the global total (Pardey and Beddow, 2013). Organizations carrying out agricultural research are increasingly being asked to show that their resources are used effectively to achieve their goals (Kelley, Ryan and Gregersen, 2008).

1.3 Outputs, outcomes and impacts

Government institutions, as well as international and national organizations, worldwide have therefore been placed under increasing pressure to ensure transparency and to provide 'value for money' when using the funds provided to them by their taxpayers or donors. In response, many of them have introduced results-based management (RBM), which is "a management strategy focusing on performance and achievement of outputs, outcomes and impacts" (OECD-DAC, 2010). In RBM, organizations try to ensure that all of their processes, products and services contribute to the achievement of desired results and the information and evidence on actual results is used to inform decision-making on the design, resourcing and delivery of activities, as well as for accountability and reporting (e.g. UNDG, 2011).

A key component of RBM practices, among others, is the sequential 'results chain' or impact pathway, in which INPUTS lead to ACTIVITIES which produce OUTPUTS leading to OUTCOMES which then lead to IMPACTS. Inputs include funds, technical assistance and human and other resources; activities include actions taken or work done; outputs include new products, services and capacities; and outcomes represent the likely or achieved short-term and medium-term effects of the outputs (UNDG, 2011). Impact, instead, refers to the long-term effects. Following the commonly-used OECD-DAC (2010) definition, impact refers to the "Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended". These effects can be economic, socio-cultural, institutional, environmental, technological or of other types.

For example, if agricultural research was carried out to develop a new vaccine against a disease of farmed fish, the outputs might be the new vaccine that was developed, the outcome would be the degree of adoption and use of the new vaccine by the fish farmers and the impacts might be the long-term effects that adoption of the vaccine had on the economic well-being, employment and livelihoods of the fish farmers and on the environment (e.g. possible reduced release of antibiotics into fish farm water).

1.4 Ex post impact assessment (epIA)

This e-mail conference is about ex post assessment of the impacts of agricultural research. The term 'ex post' refers to the fact that the assessment is done after the intervention (which is derived from agricultural research) has been completed. This is in contrast to ex ante impact assessment, where 'ex ante' refers to assessment carried out beforehand, to predict the impacts that the research-led intervention may have in the future.

Following on from the OECD-DAC definition of impact, which refers to "long-term effects produced by a development intervention" is the understanding that epIA needs to attribute impacts to interventions and does not just assess what has happened (Leeuw and Vaessen, 2009). Following Kelley, Ryan and Gregersen (2008), epIA is defined here as focusing on "changes in selected indicators of achievement of mission-level goals (e.g. income changes or sustainable poverty alleviation) that can be attributed to specific R&D interventions compared to what the indicators would have been in the absence of the interventions, that is, the counterfactual situation or control". The issue of attribution, i.e. establishing a causal link between observed changes and a specific intervention (OECD-DAC, 2010), is central in the epIA literature. Note, terminology in this field can vary considerably - see e.g. Martin and Nelson (2012) and Roger (2012). EpIA, as defined here for this e-mail conference, is commonly referred to as 'impact evaluation' elsewhere. The definition of impact evaluation, however, also varies in the literature, with White (2009) noting that, for some, it includes any evaluation which refers to impact indicators while, for others, it includes only those studies where attribution is central and a counterfactual is used.

As Leeuw and Vaessen (2009) underline, epIA is just one component of a comprehensive evaluation package. For agricultural research, this package includes ex ante impact assessment to evaluate alternative potential research options, carried out in the planning stage of a project or programme, as well as monitoring and evaluation activities, carried out while the research programme is ongoing, such as reporting on outputs and achievements in the context of plans and expectations and evaluating the quality of the research being done (Kelley, Ryan and Gregersen, 2008). EpIA of agricultural research is therefore not the same as research evaluation but instead is just one component of the much larger research evaluation package.

EpIA may serve several purposes but one which has often been espoused relates to accountability and strategic validation, i.e. "in providing empirical evidence of the effectiveness of past investment for generating outcomes of interest and validating the relevance and efficiency of overall strategies pursued" (Kelley, Ryan and Gregersen, 2008). But epIA is often undertaken with other objectives in mind, such as learning or producing relevant feedback to priority setting. According to Leeuw and Vaessen (2009), a fundamental reason for conducting epIA is "to learn about what works and what doesn't and why" and thus contribute to more effective interventions in the future.

This Background Document aims to provide information that participants will find useful for the e-mail conference. In Section 2 an overview is provided of the main approaches and methodologies for carrying out epIA of agricultural research. Section 3 discusses communication of the epIA results to donors and policy-makers. Section 4 presents some specific guidance about the topics that are to be discussed in the conference. Section 5 provides references of articles mentioned in the document, abbreviations and acknowledgements.

2. Description of the main approaches and methodologies for carrying out epIA of agricultural research

EpIA of agricultural research can be classified into two broad groups, i.e. macro-level and micro-level assessment (Walker et al., 2008; Maredia, 2009). The first looks at generic technologies/policies or agricultural research as a whole and considers, for example, the impacts of agricultural research at the regional or national level. The second looks at one or more specific technologies or policies and considers, for example, the impacts of research resulting in the adoption of a specific output such as a livestock vaccine.

Both at the macro- and the micro-level, the overwhelming focus of epIA studies has been on economic returns.

2.1 Macro-level assessments

2.1.1 General approach

Although macro-level, or aggregate, epIA does not look at specific interventions, it is important because it improves our understanding of the relative success of agriculture research in effecting macro-level impact goals and, thereby, the relative strategic merits of investing in agricultural research (Maredia, 2009). This has been a very active field over the past decades (e.g. Alston et al., 2000; Rao, Hurley and Pardey, 2012). In these kinds of assessments, the main aim is to relate previous investments in agricultural research to changes in agricultural productivity and profitability. These assessments depend on having access to reliable data on agricultural output, research expenditure and other determinants of productivity change (Walker et al., 2008). They normally use regional or national secondary data, i.e. previously existing data created for other purposes and produced by other sources such as government agencies (Creevey and Snodgrass, 2006), on agricultural research expenditures and agricultural outputs.

These assessments need i) relationships between the size of investment in agricultural research and output or productivity ii) relationships between increased productivity flows and economic benefits and iii) procedures to account for the time lags (which can be substantial) between the original investment in agricultural research and the eventual reaping of economic returns on that investment (Alston et al., 2000).

The economic impacts are typically expressed using one or more commonly-used and related parameters. Before defining them, let us consider that it is common sense that money received (or spent) in the future is worth less than the same amount of money received (or spent) now. To account for this in economic analysis the concept of discounting is used, where the value of a cost incurred, or benefit received, is adjusted using a pre-determined annual percentage, i.e. the discount rate, to obtain its present value. For example, with a discount rate of 10%, 100 euro received in seven years time is worth roughly half what it would be worth now. The 'stream' of economic benefits arriving in different years can therefore all be expressed in present value terms. So also can the stream of research investments that are made over different years.

A common parameter used in many epIAs is the net present value (NPV), which is the present value of the economic benefits minus the present value of the costs. Another commonly-used parameter is the benefit-cost ratio (BCR) which is the ratio of the present value of benefits to the present values of costs. Thirdly, there is the internal rate of return (IRR), which is the discount rate at which the NPV is zero.

Alston et al. (2000) note that while the NPV is usually regarded as the best parameter for studying economic rates of return, the IRR is the most commonly estimated parameter in studies of this nature. Indeed, in their landmark analysis of published studies of rates of return, Alston et al. (2000) were able to use data from nearly 300 publications with over 1,800 estimates of IRR published between 1953 and 1997. Roughly 80% of the estimates were from ex post and 20% from ex ante assessments. Their analysis showed that IRR estimates were affected by a number of factors, and that they were:
- Higher when the IRR was nominal (i.e. not adjusted for inflation) than real (adjusted for inflation)
- Higher when ex post than ex ante
- Higher for research on field crops and lower for natural resources research
- Lower when the assessment was carried out by someone from the university or the private sector than from the government
- Lower when the results were published in a refereed journal than a non-refereed source
- Lower with a longer research gestation lag (i.e. the initial time period when there are research costs but no benefits).

This continues to be an active area of research. Rao, Hurley and Pardey (2012) included a further 77 studies published in 1999-2011 to those of Alston et al. (2000) and carried out an analysis of 2,186 assessments published in 359 studies between 1958 and 2011. Among this huge dataset, the IRR was the main measure of economic returns to agricultural research (reported in 95% of studies), 26% of the studies reported the BCR and 21% reported both parameters.

2.1.2 Methodology for macro-level economic impact assessments

Alston et al. (2000) showed that two models dominate. About 40% of the estimates in their study were calculated using econometric models and about 60% from economic surplus models.

The econometric approach employs instruments such a production function (which relates outputs to factors of production) to estimate changes in productivity due to investments in research (Maredia, Byerlee and Anderson, 2000). Typical models (e.g. Thirtle et al., 1998; Thirtle, Lin and Piesse, 2003) use an equation which says that changes in output (yield) are explained by changes in conventional inputs (such as land, labour, machinery, fertilizer), non-conventional inputs (such as education level) and research (normally expressed as financial investments in research adjusted for the time lag between research investments and yield changes). From the model, the marginal contribution of research investments to aggregate production can then be estimated. The econometric model requires good quality time series data (i.e. recorded over several different years), a requirement that can often be difficult to fulfill below the national or state level in many developing countries (Maredia, Byerlee and Anderson, 2000).

The economic surplus approach, on the other hand, does not present similar data challenges and is used more often to estimate the economic impacts of agricultural research. The approach is used within a partial equilibrium framework, i.e. where the focus is on one sector of the economy, such as the agricultural or crop sector, and where most other economic variables are assumed to be constant (the alternative is the general equilibrium model which considers the whole economy).

The economic surplus model is based on the idea that research leads to outputs (such as new technologies) whose use increases the quantities of agricultural produce per year, leading to a shift in the supply curve and a reduction in price. The method allows the economic benefits (the economic surplus) to be estimated as well as to measure the relative benefits for farmers and consumers. The key parameters needed to estimate the economic benefits are the initial price and quantity of the agricultural produce; the price elasticity of the demand and supply functions; and the direct effect of technological change on the supply curve, i.e. the so-called 'k factor'. To estimate the k factor, estimates of yields per hectare (or another relevant variable) are needed from experimental or field data. Depending on the sector and the country being considered, different kinds of economic surplus models can be used. These might include models of a closed economy (where international trade is negligible), a small country in an open economy or a large country in an open economy - each of which has implications for the effect on prices. See Alston, Norton and Pardey (1995) or Maredia, Byerlee and Anderson (2000) for more details.

2.1.3 Assessment of non-economic impacts

As mentioned earlier, most epIA studies have typically focused on the impacts of agricultural research on economic returns. However, increasing numbers of studies are aiming to uncover the impacts on other dimensions such as the environment, natural resources management, human health or nutrition, food security or poverty, gender issues or government policy. While the economic rates of return to investing in agricultural research may also be estimated in these studies, they are not the ultimate goal but are rather the means to estimate the other effects (Walker al al., 2008).

There is increasing pressure to measure the impacts of agricultural research on these non-economic dimensions. Apart from their inherent importance, another reason is the increased focus on internationally agreed development goals. Most important of these are the eight Millennium Development Goals, where governments in 2000 committed themselves to quantified targets for addressing poverty, hunger, disease, lack of adequate shelter and exclusion, while promoting gender equality, education and environmental sustainability. As they will expire in 2015, the UN Secretary-General and UN member states have launched a number of processes to develop a new set of "post-2015" goals, called Sustainable Development Goals (SDGs). Currently, there are 10 SDGs proposed, including to end extreme poverty including hunger; improve agriculture systems and raise rural prosperity; and secure ecosystem services and biodiversity, and ensure good management of water, oceans, forests and natural resources (http://unsdsn.org/).

Until recently, few multi-dimensional impact assessments were carried out because of the challenges the non-economic dimensions present at the conceptual and methodological level as well as regarding data availability and the fact that the field and methods of quantitative impact assessment have mainly been developed by economists (Kelley, Ryan and Gregersen, 2008).

Nevertheless, this does seem to be changing and increasing numbers of reports are focusing on such issues. For example, Alene et al. (2009) studied the impacts that public investments in maize research had on poverty in West and Central Africa from 1981 to 2005. Secondary data from FAOSTAT (on maize area, yields, production and prices), the World Bank (on poverty) and national sources and the International Institute of Tropical Agriculture (on investments in maize research) were used. They first used the economic surplus model to calculate the change in economic surplus due to maize research and thereafter used this result to estimate the number of poor people that were lifted out of poverty due to maize research.

2.2 Micro-level assessments

Whereas macro-level impact assessments typically use secondary data sources and are looking at the impacts of agricultural research at the 'big picture' level (e.g. the whole agricultural sector, or part thereof, at the regional or national level), micro-level, or disaggregate, assessments tend to use primary data (i.e. collected by researchers during the research project) and focus on the impacts of one or more well-identified technologies produced as a result of investments in agricultural research (Walker et al., 2008).

As for macro-level epIA, most studies of micro-level epIA have focused on economic rates of return. According to Walker et al (2008), such studies are "one of the staple tasks of practitioners who carry out epIAs on agricultural research". As for macro-level economic impact assessments, econometric and economic surplus methods have been frequently used.

A wide range of other methodologies are also available which have been used for assessing impacts in other areas, such as education, human health and the social sciences, and are now beginning to be applied to epIA of agricultural research (de Janvry et al., 2011). The methodologies can be grouped into those that are quantitative, qualitative or mixed, i.e. which combine quantitative and qualitative approaches (Leeuw and Vaessen, 2009).

While the distinction between quantitative and qualitative methods is not always straightforward (see e.g. discussion in Stern et al., 2012), quantitative methods are commonly defined as those which produce data in the form of numbers while qualitative methods tend to produce data that are stated in prose or textual forms (Garbarino and Holland, 2009). For example, Creevey and Snodgrass (2006) state that "quantitative methods are research methods dealing with numbers and anything that is measurable. They typically involve numerical tabulations and statistical comparisons made possible by systematic surveys, observations, or analysis of records. In contrast, qualitative methodologies aim to understand, report, and evaluate the meaning of events for people in particular situations. The focus of qualitative methodologies is the way in which participants (rather than the researcher) interpret their experiences and construct reality. Common examples include unstructured or semi-structured interviews, focus group discussions, open ended questionnaires, case studies, and participant observation".

2.2.1 Quantitative methods

According to Leeuw and Vaessen (2009), quantitative methods can have a comparative advantage in addressing the issue of attribution. The counterfactual is simulated by comparing the situation of the "treatment" group, affected by the intervention (in our case, agricultural research leading to e.g. a new animal vaccine), with the situation of an equivalent "control" group that is not affected by the intervention. A key issue these methods need to tackle is 'selection bias', where those in the treatment group are different in some way from those in the control group. The underlying idea of selection bias is that farmers who choose to adopt a given research-derived innovation might differ from those who do not adopt it. For example, farmers who choose to adopt a new livestock vaccine might be, at the same time, more skilled farmers, who have greater access to farm credit, have genetically superior animals etc. All this might mean that, even without adopting the vaccine, their livestock are likely to produce more than those from the group of farmers who choose not to adopt the vaccine. By comparing the yields of livestock in the two groups, estimates of the effect of the vaccine are confounded with those of the effect of all the other characteristics that differ between the two groups of farmers.

The many quantitative methods can be broadly sub-divided into experimental and quasi-experimental methods:

Experimental methods

Randomized controlled trials (RCTs) are the norm in many biological and pharmaceutical fields, where individuals are randomly assigned to the treatment or control group and a simple statistical test for the difference between the two group means after the intervention is sufficient to estimate its effect. As assignment of individuals to the two groups is random, selection bias is minimized as long as random assignment is followed (i.e. every farmer who is offered the research-derived innovation adopts it, and no-one in the control group has an alternative access to it) and as long as assignment to the treatment or control groups does not change the farmers' behavior. That such issues can influence the value of RCTs was shown, for example, by Bulte et al. (2014) who found that the behaviour of Tanzanian farmers in a cowpea RCT seemed to change depending on whether they received seed from traditional or modern cowpea varieties, thus threatening the validity of the RCT.

To illustrate how RCTs might work in practice, consider that a research programme was carried out to develop a new vaccine against a given livestock disease. Then a set of farmers would be chosen at random who are given access to the vaccine on their farms while another set would be chosen at random to act as a control, continuing to raise livestock without using the vaccine. By comparing the production of livestock in the two groups, estimates of the effects of the vaccine on production are derived.

The drive for more evidence-based and rigorous impact assessment, particularly by donors, has meant that RCTs have been championed in a wide range of areas, such as development assistance and social protection programmes (Devereux et al., 2013), and this has led to their role becoming quite contentious in some cases. Also in the agricultural research community, it is a topic for debate (Martin and Nelson, 2012). For example, de Janvry et al. (2011) provide a good discussion of the advantages and disadvantages of RCTs for assessing the impact of research-induced technological change in crops.

Quasi-experimental methods

Rather than randomly allocating farmers to the treatment or control groups, quasi-experimental methods operate in the reality where the farmers themselves have made the decision to adopt (or not) the research-derived innovation. As the quasi-experimental methods intervene after these decisions have been made, selection bias can be a real problem. To overcome it, quasi-experimental methods aim to create an appropriate control group for comparison with the treatment group. Several such methods exist. For example, in propensity score matching (PSM) a control group is created by selecting farmers on the basis of observable and relevant characteristics that are similar to those of the treatment group (Leeuw and Vaessen, 2009). Matching is done using the propensity score, i.e. the predicted probability of being included in the treatment group given observable characteristics (e.g. farm size). Although a number of recent impact studies in agriculture have used PSM, de Janvry et al. (2011) note some statistical weaknesses of the method related to the fact that it does not solve the potential selection bias that results from not accounting for unobservable (not recorded) characteristics, such as farmer skills or access to credit, that may differ between the treatment and control groups.

A potential solution can be found when, in addition to data on the treatment and control groups, data is also collected before and after the intervention for each group. In this case, it is possible to use the difference-in-difference (DD, also called double difference) method where, in addition to measuring the difference (D1) between the treatment and control groups after the intervention, the differences (D0) between the two groups are also measured before the intervention. By using the difference between these two (i.e. D1-D0), the problem of selection bias from unobservable characteristics can be dealt with, assuming the unobservable characteristics evolve in the same way over time between the two groups. In a number of cases, the DD method has also been used with PSM for agricultural research studies (de Janvry et al., 2011)

2.2.2 Qualitative methods

An important criticism raised against quantitative methods for impact assessment is that they focus on whether or not there is an impact, but do not address the question of how and why the effects arise. So, instead of trying to quantify the effects attributable to an intervention, qualitative methods give more emphasis to "opening the black box", seeking to understand the mechanisms which lead to effects. This may be advantageous in many situations, where some or all of the direct or indirect, intended or unintended outcomes and impacts of the research may not be recorded - because it is impossible to measure all potential indicators of impact and because some indicators are not easily quantifiable (CGIAR, 2008).

As advocated in Walker et al. (2008) and Leeuw and Vaessen (2009), evaluation should start with the preliminary step of establishing the underlying theory of the project, its theory of change (also called theory of action, or programme theory), i.e. make explicit the mechanisms through which the project stakeholders expect the impact to arise. This is done in impact pathway analysis (Douthwaite et al., 2003), where one seeks to describe in detail the main steps in the overall pathway from research inputs to outputs to outcomes to impacts (see Walker et al., 2008, who also provide some examples). This "pathway identification" step can be done in various ways, based on the theory and empirical literature, or through participatory approaches. Outcome mapping is another method to identify the causal pathway, which however deals with the problem of how to attribute impact by putting the emphasis on results achieved "upstream" from impact. It focuses on outcomes as behavioural change, where outcomes are defined as changes in the behaviour, relationships, activities or actions of the people, groups and organizations with whom a programme works directly (Earl, Carden and Smutylo, 2001).

After pathway identification, the next step is to evaluate the impact pathway, investigating whether the highlighted mechanisms occurred as expected. The outcome mapping approach suggests to do this step using contribution analysis (Mayne, 2012), which assesses the programme's contribution by carefully seeking evidence and possible counter-arguments for every step of the chain. Another approach is to evaluate the impact pathway answering counterfactual questions through qualitative methods.

Unlike quantitative methods where counterfactuals are obtained from large control groups (chosen at random or not), qualitative methods will involve stakeholders answering the same hypothetical question of how the identified outcomes and mechanisms would have evolved if the programme had not taken place. A large range of qualitative methods can be used for this purpose, such as key informant interviews, where interviews are carried out with people considered to have special knowledge or expertise because they are experts in a given professional area or because of their position in the community or organization, or focus groups, where discussions by a group of people with a common interest are moderated so that they focus on the topic of concern (FAO, 2013).

Another complementary approach is social network analysis (SNA), which provides a visual and quantitative representation of the linkages between different people relevant to the research project to understand the formal and informal networks that are operating (Hovland, 2007). This helps to identify key actors who significantly influence the links in the causal pathway, particularly the links related to the dissemination of information, a key component of technology diffusion. For example, CGIAR (2008) includes a case study of a project on community-based fisheries management in Bangladesh, where SNA was used to identify the key players at various levels (national, intermediate and local) involved in decision-making and transmission of information and knowledge (through policy briefs, reports, discussion groups etc.). This helped evaluate one of the steps of the programme's impact pathway (awareness about community-based management), getting in-depth knowledge about how this process occurred.

2.2.3 Mixed methods

The individual strengths of quantitative and qualitative methods have led to calls to combine them in so-called mixed methods (e.g. Bamberger, 2012). Thus, Leeuw and Vaessen (2009) argue that "A mix of methods - triangulating information from different approaches - can be used to assess different facets of complex outcomes or impacts, yielding greater validity than from one method alone". Similarly, Stern et al. (2012) state there is widespread agreement about the benefits of combining quantitative and qualitative methods. Case studies of applications of mixed methods are presented in Leeuw and Vaessen (2009); Bamberger, Rao and Woolcock (2010); and Bamberger (2012). For example, Leeuw and Vaessen (2009) describe the work done to assess the impact of International Fund for Agricultural Development (IFAD) projects in Gambia, Ghana and Morocco, where a quantitative survey of 200-300 households (including both project and control groups) was carried out in addition to focus group discussions and key informant interviews.

Overall, there seems to be growing recognition that qualitative methods can make an important contribution to epIA either on their own or as a support to quantitative methods. However, the relative importance of quantitative versus qualitative methods is still the subject of much debate. For example, Devereux et al. (2013) highlight the limitations of quantitative methods, including RCTs, and argue for greater emphasis to be given to qualitative and mixed methods for evaluating the impacts of social protection programmes. De Janvry et al. (2011), on the other hand, while recognising that qualitative methods are useful, conclude that "When the goal is to quantify impacts with any level of accuracy, qualitative methods must play a secondary role to that of careful quantitative analysis".

2.2.4 Policy-oriented research

When thinking about the outputs of agricultural research, it is important to consider that they can encompass far more than new products (such as vaccines or crop varieties) or technologies/practices. For example, the research project might involve learning new scientific knowledge or management skills, resulting in capacity development at the individual or organizational level, and use of this capacity might result in important impacts in other areas in the future (Davis et al., 2008).

Research may also be carried out to assist policy-making, so one area of interest in epIA is to study the economic impacts of specific policies introduced as a result of this kind of agricultural research. Following Walker, Ryan and Kelley (2010), policy-oriented research can be defined as that which "aims, at least in part, at affecting choices made by governments or other institutions whose decisions are embodied in laws, regulations, or other activities that generate benefits and costs for people who are affected by those governments or institutions". For example, the Syrian government limited allocations of fertilizer to barley because of the perception that it was not profitable to apply inorganic fertilizer to the crop. Research was then carried out to investigate this issue, which provided clear evidence of economic response to moderate levels of fertilization of barley in arid agriculture. This resulted in changes to the Syrian national fertilizer policy which in turn resulted in substantial economic gains being estimated for both producers and consumers (Ahmed, Shideed and Mazid, 2010). Whereas technology-based studies tend to use quantitative methods only, assessments of policy-oriented research tend to use mixed models (Walker et al., 2008).

2.2.5 Assessment of non-economic impacts

As discussed in Section 2.1.3, although most of the epIA work continues to focus on economic impacts, far greater attention is now given to assessment of the impacts of agricultural research on non-economic dimensions, including the development of appropriate methodologies and approaches to do so (Maredia et al., 2014). In doing so, the economic impacts are often assessed as an intermediate step. For non-economic impacts, mixed methods tend to be used and collection of relevant data tends to be both time-consuming and costly (Walker et al., 2008).

Non-economic dimensions of particular interest include poverty alleviation and food security. For example, Adato and Meinzen-Dick (2007) describe five case studies where impacts on poverty were assessed at household and community levels using the 'livelihoods approach' while Maredia et al. (2014) describe some recent studies assessing impact on poverty, inequality and food security.

The environmental impacts of agricultural research are also of major interest. As noted by Renkow (2011) in his review of Consultative Group on International Agricultural Research (CGIAR) research in this area, there may be environmental impacts associated with changes attributable to the adoption of technologies or policies resulting from agricultural re¬search and these impacts may be positive or negative, intended or unintended, and may be felt on-farm, locally or globally. He found, however, that only a small number of CGIAR epIA studies had studied environmental impacts. Bennett (2011) described four case studies where economic values of the environmental impacts were estimated, noting however the challenges regarding data availability and limited expertise in applying such methods for environmental impacts.

3. Communicating the epIA results to donors and policy-makers

While, as noted in the Introduction, there is a major drive for organizations to assess the impacts of agricultural research, and to broaden the coverage of effects to be considered and to improve the methodologies used for this purpose, there seems to be relatively little information available about how the eventual results of the epIA studies are communicated to donors, policy-makers or other relevant stakeholders and how they then use the results.

A recent survey of impact assessment policies and practices of the European Initiative for Agricultural Research for Development (EIARD) countries analysed responses from 16 countries plus the European Commission (Martin and Nelson, 2012). The survey found that, in most cases, no information was provided about dissemination and communication of the evaluation results. Also, they found no evidence that creative methods, e.g. involving video materials, were used to promote dialogue between end beneficiaries and those commissioning the assessment. Raitzer and Winkel (2005) presented the results of interviews, discussion and a survey involving CGIAR donors, carried out to examine their impact assessment needs and uses. From the interviews they reported that, because of time constraints, there was widespread demand from donors for brief summaries of impact assessment results. As a consequence, the CGIAR Standing Panel on Impact Assessment (SPIA) publishes periodic Impact Briefs, summarizing the main findings of specific epIA studies (http://impact.cgiar.org/impact-briefs).

Similar issues with donor constraints were highlighted by Norton and Alwang (2012) who presented the results of informal consultations with seven donor organisations about the mechanisms they use to prioritise investments in agricultural research and development. Most of them reported using only informal mechanisms, explaining that they lacked capacity, resources or staff time to conduct formal, technical priority setting exercises. Some donors with more technical staff, such as the United States Agency for International Development (USAID) and UK Department for International Development (DFID), sometimes engaged in formal priority setting but not on a regular basis. The consultations also found that many donors expressed a need for more information to inform their priority setting. They also wanted more information on the impact of agricultural research on non-efficiency objectives, such as health, nutrition and environmental sustainability.

How do the donors and policy-makers use the epIA results? Martin and Nelson (2012) reported that there are only a few examples where donors provide publicly available feedback on the assessments that they have commissioned, but suggested the practice might become more common in the future. From their survey, they found there was little information available on how impact assessment findings had been taken up and implemented. For the CGIAR, interviews revealed that factors such as political priorities, perceived scientific quality and desire for continuity often influenced funding decisions more than epIA results. However, the epIA results seemed to inspire confidence among the donors, thus playing an important role in maintaining CGIAR credibility and ensuring continued support for its work (Raitzer and Winkel, 2005; Raitzer and Kelley, 2008).

4. Main questions to be discussed by participants in the conference

This e-mail conference enables participants from around the world to share and discuss their experiences, lessons learned and perspectives regarding the concepts, approaches and methodologies for ex post impact assessment of agricultural research. Note that the term 'agricultural research' encompasses research in the crop, livestock, forestry, fishery and aquaculture sectors, so the conference covers epIA relating to any of these sectors. Note also that the aim of the conference is not to discuss the potential positive or negative impacts of any specific technologies or products derived from agricultural research. Instead, the conference focuses on the concepts, approaches and methodologies used for epIA of agricultural research.

The main kinds of topics to be discussed by participants in the conference are described below:

4.1 Approaches and methodologies for epIA

As seen in Section 2, for epIA of agricultural research, both at the macro- and micro-level, a range of approaches and methodologies are available, which have different statistical properties, data requirements and practical characteristics.

- What have been participants' experiences and lessons learned from applying different methodologies (including general issues such as what have been the difficulties in using them?; how was the complexity of the processes from inputs to impacts dealt with?; what methods were used to identify the beneficiaries of the impacts?; how were reliable and good-quality indicators identified?)
- Also, for the future, what perspectives do the participants see for the different methodologies?

4.2 Assessment of non-economic impacts

As seen in Sections 2.1.3 and 2.2.5, for both macro- and micro-level epIA respectively, there is increasing interest in assessing the impacts of agricultural research on environmental (including natural resources management and sustainability), social (including poverty and hunger alleviation), government policy and other non-economic dimensions, although these assessments can represent considerable data and methodological challenges.

- What cost- and time-effective approaches and methodologies can be used to best address the challenges of assessing non-economic impacts?
- What perspectives are there that epIA studies of non-economic impacts will become easier and more commonplace in the future?

4.3 Quantitative versus qualitative methods

As described in Section 2.2, quantitative and qualitative methods have different strengths and weaknesses and their deployment involves use of different skills by the people carrying out the epIA. The importance of their relative roles in impact assessment is also a topic of considerable debate.

- For epIA of agricultural research, what are the relative merits of the two kinds of methods?
- When are qualitative methods to be preferred?
- When are mixed methods, combining quantitative and qualitative, to be preferred (considering also the issue of cost of carrying out the assessment)?
- Are there quantitative or qualitative methods currently used for epIA in other areas which might be particularly useful for epIA of agricultural research?

4.4 Randomised controlled trials (RCTs) versus other methods (quantitative or qualitative)

As described in Section 2.2.1, the drive for more evidence-based and rigorous impact assessment, particularly by donors, has meant that use of RCTs has been championed in a wide range of areas and their potential role is also of considerable interest for epIA of agricultural research.

- How important are RCTs likely to be for epIA of agricultural research and when are they likely to be particularly advantageous?
- How important, cost- and time-effective are epIA studies from RCTs compared to those from other quantitative or qualitative methods, including qualitative narrative stories?

4.5 EpIA in the different food and agricultural sectors

As seen from Alston et al. (2000), the overwhelming focus of epIA studies in the past has been on field crops, with about 1,000 estimates of economic rates of return found in the literature compared to over 300 for livestock and less than 100 for research with a natural resources focus, including forestry and fisheries. According to Raitzer and Kelley (2008), the uneven coverage of epIA work across the different food and agricultural sectors might be one of the reasons why epIA results are not crucial for donors when deciding on research funding allocations.

- Is the uneven focus of epIA work across the crop, livestock, forestry, fishery and agro-industry sectors an important issue?
- If so, what incentives can be provided to ensure that all the food and agricultural sectors are better covered?
- To what degree can epIA methods developed for one sector be transferred to others?

4.6 Cost-effective epIA

As described in the Introduction, organizations carrying out agricultural research are under increased pressure to assess the impact of their research activities. There is also a drive to measure indicators that are not purely of an economic nature and to carry out the epIA work in a more comprehensive and rigorous fashion, which may require gathering data on several relevant indicators and using more than one method (e.g. combining qualitative and quantitative methods). This kind of work entails use of financial and human resources which might otherwise be used for other purposes, such as carrying out research. For example, in their book with case studies of the impact of agricultural research on poverty, Adato and Meinzen-Dick (2007) indicate that each case study costed around 200,000 USD. While noting that the sum is relatively small compared to the total research budget, they say "the challenge now is to evolve and adopt cost- and time-effective approaches and methods...".

- How can meaningful and high-quality epIA studies be carried out which do not cost too much money?
- What is the appropriate proportion of a research budget to spend on epIA?
- With very limited research funding, is it worthwhile dedicating resources to epIA?
- What is the minimum budget one needs to carry out a meaningful epIA of agricultural research?
- Are there good examples of cost- and time-effective approaches and methods for epIA of agricultural research?

4.7 Communicating the epIA findings

As seen in Section 3, limited information seems to be available about how the final results of epIA studies are communicated to donors or policy-makers and how they then use these results.

- What have been participants' experiences and lessons learned from communicating the epIA findings to donors or policy-makers?
- What is the best approach and format for presenting the findings to them?
- How can it be ensured that the results will be read and considered by donors and policy-makers?
- What examples are available which demonstrate that the results of epAI studies have been used in a policy process or to make strategic funding decisions?

4.8 EpIA as one component in the overall evaluation package

The primary purpose and role of epIA of agricultural research has been critically examined on occasions (e.g. Ekboir, 2003) and continues to be a topic of current debate.

- What is the primary purpose of epIA of agricultural research today?
- Compared to the other components in the overall evaluation package, how important is epIA today and how important is it likely to be in the future?

4.9 Instructions for sending a message

Before submitting a message to the e-mail conference, participants are requested to:
a) Ensure that it addresses the topics mentioned in Section 4 above.
b) Limit its length to a maximum of 600 words.
c) Follow the 'Guidelines for Sending Messages' contained at the end of the Welcome Text that participants receive when they subscribe to the conference. Among other things, the Guidelines note that participants: are assumed to be speaking on their own behalf and not on behalf of their employers (unless they indicate otherwise); should introduce themselves briefly in their first posting to the conference, providing also their full work address at the end of the message; and may not post libellous, insulting or defamatory messages or materials, or links to such materials and should exercise tolerance and respect toward other participants whose views may differ from your own.

5. References, abbreviations and acknowledgements

Adato, M. and R. Meinzen-Dick (eds.). 2007. Agricultural research, livelihoods, and poverty: Studies of economic and social impacts in six countries. International Food Policy Research Institute. http://www.ifpri.org/publication/agricultural-research-livelihoods-and-poverty

Ahmed, M.A.M., Shideed, K. and A. Mazid. 2010. Returns to policy-oriented agricultural research: The case of barley fertilization in Syria. World Development 38: 1462-1472. http://www.sciencedirect.com/science/journal/0305750X/38/10

Alene, A.D., Menkir, A., Ajala, S.O., Badu-Apraku, A.S., Olanrewaju, V., Manyong, M. and A. Ndiaye. 2009. The economic and poverty impacts of maize research in West and Central Africa. Agricultural Economics 40: 535-550.

Alston, J.M., Norton, G.W. and P.G. Pardey. 1995. Science under scarcity: Principles and practice for agricultural research evaluation and priority setting. CAB International: Wallingford, UK.

Alston J.M., Chan-Kang C., Marra M.C., Pardey P.G., and T.J. Wyatt. 2000. A meta-analysis of rates of return to agricultural R&D: Ex pede herculem? IFPRI Research Report 113. International Food Policy Research Institute. http://www.ifpri.org/sites/default/files/publications/rr113.pdf (750 KB).

Bamberger, M. 2012. Introduction to mixed methods in impact evaluation. Impact Evaluation Guidance Note 3. InterAction. http://bit.ly/TiAvrQ

Bamberger M., Rao V. and M. Woolcock. 2010. Using mixed methods in monitoring and evaluation: Experiences from international development. Policy Research Working Paper 5245. World Bank. http://elibrary.worldbank.org/doi/pdf/10.1596/1813-9450-5245 (650 KB)

Beintema, N.M., Stads, G-J., Fuglie, K. and P. Heisey. 2012. ASTI global assessment of agricultural R&D spending: Developing countries accelerate investment. International Food Policy Research Institute. http://www.asti.cgiar.org/globaloverview

Bennett, J. 2011. Advancing ex-post impact assessment of environmental impacts of CGIAR research: Conceptual issues, applications and the way forward. CGIAR Independent Science and Partnership Council (ISPC). http://www.sciencecouncil.cgiar.org/fileadmin/templates/ispc/documents/Publications/EIAStudy2011James.pdf (1 MB)

Bulte, E., Beekman, G., Di Falco, S., Lei, P. and J. Hella. 2014. Behavioral responses and the impact of new agricultural technologies: Evidence from a double-blind field experiment in Tanzania. American Journal of Agricultural Economics 96: 813-830.

CGIAR. 2008. Impact assessment of policy-oriented research in the CGIAR: Evidence and insights from case studies. CGIAR Independent Science and Partnership Council (ISPC). http://www.sciencecouncil.cgiar.org/fileadmin/templates/ispc/documents/Impact_Assessment/SC_IA_PORIA2008.pdf (1.1 MB)

Creevey, L. and D. Snodgrass. 2006. Collecting and using data for impact assessment. Impact Assessment Primer Series Publication 3. United States Agency for International Development (USAID). http://pdf.usaid.gov/pdf_docs/PNADN197.pdf (200 KB).

Davis J., Gordon J., Pearce D. and D. Templeton. 2008. Guidelines for assessing the impacts of ACIAR's research activities. ACIAR Impact Assessment Series Report No. 58. http://aciar.gov.au/files/node/10103/ias58_pdf_20268.pdf (1.7 MB).

de Janvry, A., Dustan, A. and E. Sadoulet. 2011. Recent advances in impact analysis methods for ex-post impact assessments of agricultural technology: Options for the CGIAR. CGIAR Independent Science and Partnership Council (ISPC). http://areweb.berkeley.edu/~sadoulet/papers/deJanvryetal2011.pdf (1.1 MB)

Devereux, S., Roelen, K., Béné, C., Chopra, D., Leavy, J. and J.A. McGregor. 2013. Evaluating outside the box: An alternative framework for analysing social protection programmes. IDS Working Paper 431. Institute of Development Studies. http://www.ids.ac.uk/publication/evaluating-outside-the-box-an-alternative-framework-for-analysing-social-protection-programmes

Douthwaite, B., Kuby, T., van de Fliert, E. and S. Schulz. 2003. Impact pathway evaluation: an approach for achieving and attributing impact in complex systems. Agricultural Systems 78: 243-265.

Earl, S., Carden, F. and T. Smutylo. 2001. Outcome mapping: Building learning and reflection into development programs. International Development Research Centre. http://web.idrc.ca/openebooks/959-3/

Ekboir, J. 2003. Why impact analysis should not be used for research evaluation and what the alternatives are. Agricultural Systems 78: 166-184. http://www.sciencedirect.com/science/journal/0308521X/78/2

FAO. 2013. Qualitative research of the economic impacts of cash transfer programmes in sub-Saharan Africa: A research guide for the From Protection to Production project. http://www.fao.org/fileadmin/user_upload/p2p/Publications/Qualitative_guide_PtoP.pdf (1.3 MB)

Garbarino, S. and J. Holland. 2009.  Quantitative and qualitative methods in impact evaluation and measuring results: Issues paper. Governance and Social Development Resource Centre. http://www.gsdrc.org/docs/open/EIRS4.pdf (500 KB).

Hovland, I. 2007. Making a difference: M&E of policy research. ODI Working Paper 281. http://www.odi.org.uk/sites/odi.org.uk/files/odi-assets/publications-opinion-files/2426.pdf (800 KB)

Kelley, T., Ryan, J. and H. Gregersen. 2008. Enhancing ex post impact assessment of agricultural research: The CGIAR experience. Research Evaluation 17: 201-212. http://impact.cgiar.org/sites/default/files/images/Kelleyetal2008.pdf (200 KB)

Leeuw, F. and J. Vaessen. 2009. Impact evaluations and development: NONIE guidance on impact evaluation. Network of Networks for Impact Evaluation (NONIE). http://siteresources.worldbank.org/EXTOED/Resources/nonie_guidance.pdf (1.9 MB)

Maredia, M.K., Byerlee, D. and J. Anderson. 2000. Ex post evaluation of economic impacts of agricultural research programs: A tour of good practice. Paper presented to the workshop on "The future of impact assessment in CGIAR: Needs, constraints, and options", Standing Panel on Impact Assessment (SPIA) of the Technical Advisory Committee, Rome, May 3-5, Rome. http://impact.cgiar.org/pdf/99.pdf (200 KB)

Maredia, M.K. 2009. Improving the proof: Evolution of and emerging trends in impact assessment methods and approaches in agricultural development. International Food Policy Research Institute. http://www.ifpri.org/sites/default/files/publications/ifpridp00929.pdf (1.9 MB)

Maredia, M.K., Shankar, B., Kelley, T.G. and J.R. Stevenson. 2014. Impact assessment of agricultural research, institutional innovation, and technology adoption: Introduction to the special section. Food Policy 44: 214-217.

Martin, A. and V. Nelson, 2012. Impact assessment policies and practices of EIARD members. European Initiative for Agricultural Research for Development. http://www.eiard.org/media/uploads/File/documents/thematic_studies/RevisedImpact%20assesment%20Final%202%20with%20logos.pdf (1.2 MB)

Mayne, J. 2012. Contribution analysis: Coming of age? Evaluation 18: 270-280.

Norton, G.W. and J. Alwang. 2012. Donor methods to prioritise investments in agricultural research and development. Synthesis Paper. Global Donor Platform for Rural Development. http://www.donorplatform.org/agriculture-research-for-development/latest/816-platform-synthesis-paper-donor-methods-to-prioritise-investments-in-agricultural-research-and-development.html

OECD-DAC. 2010. Glossary of key terms in evaluation and results based management. Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC). http://www.oecd.org/dac/evaluation/2754804.pdf (600 KB)

Pardey, P.G. and J.M. Beddow. 2013. Agricultural innovation: The United States in a changing global reality. The Chicago Council on Global Affairs. http://www.thechicagocouncil.org/UserFiles/File/GlobalAgDevelopment/Report/Agricultural_Innovation_Final.pdf (3.7 MB)

Raitzer, D.A. and T.G. Kelley. 2008. Assessing the contribution of impact assessment to donor decisions for international agricultural research. Research Evaluation 3:187-199.

Raitzer, D.A. and K. Winkel. 2005. Donor demands and uses for evidence of research impact - the case of the Consultative Group on Agricultural Research (CGIAR). CGIAR Independent Science and Partnership Council (ISPC). http://www.fao.org/docs/eims/upload/198900/Donor%20Demands.pdf (700 KB).

Rao, X. Hurley, T.M. and P.G. Pardey. 2012. Recalibrating the reported rates of return to food and agricultural R&D. University of Minnesota Department of Applied Economics Staff Papers. http://ageconsearch.umn.edu/bitstream/135018/2/RaoEtAl2012_v2.pdf (800 KB).

Renkow, M. 2011. Assessing the environmental impacts of CGIAR research: Toward an analytical framework. CGIAR Independent Science and Partnership Council (ISPC). http://www.sciencecouncil.cgiar.org/fileadmin/templates/ispc/documents/Publications/EIAStudy2011James.pdf (1 MB)

Roger, P.J. 2012. Introduction to impact evaluation. Impact Evaluation Guidance Note 1. InterAction. http://bit.ly/AwYbT1

Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R. and B. Befani. 2012. Broadening the range of designs and methods for impact evaluations. Department for International Development Working Paper 38. http://r4d.dfid.gov.uk/Output/189575/

Thirtle, C., Townsend, R.F., Amadi, J., Lusigi, A. and J. van Zyl. 1998. The rate of return on expenditures of the South African Agricultural Research Council (ARC). Agrekon 37: 612-622.

Thirtle, C., Lin, L. and J. Piesse. 2003. The impact of research-led agricultural productivity growth on poverty reduction in Africa, Asia and Latin America. World Development 31: 1959-1975.

UNDG. 2011. Results based management handbook: Harmonizing RBM concepts and approaches for improved development results at country level. United Nations Development Group. http://www.undg.org/index.cfm?P=1626

Walker, T., Maredia, M., Kelley, T., La Rovere, R., Templeton, D., Thiele, G. and B. Douthwaite. 2008. Strategic guidance for ex post impact assessment of agricultural research. CGIAR Independent Science and Partnership Council (ISPC). http://www.sciencecouncil.cgiar.org/fileadmin/user_upload/sciencecouncil/Impact_Assessment/SC_epIA_low-res_for_web.pdf (0.5 MB)

Walker, T., Ryan, J. and T.G. Kelley. 2010. Impact assessment of policy-oriented international agricultural research: Evidence and insights from case studies. World Development 38: 1453-1461. http://www.sciencedirect.com/science/journal/0305750X/38/10

White, H. 2009. Some reflections on current debates in impact evaluation. International Initiative for Impact Evaluation (3ie) Working Paper 1. http://www.3ieimpact.org/media/filer/2012/05/07/Working_Paper_1.pdf (900 KB).

****************
ABBREVIATIONS: BCR = Benefit-cost ratio; CGIAR = Consultative Group on International Agricultural Research; EpIA = Ex post impact assessment; FAO = UN Food and Agriculture Organization; IMPRESA = Impact of Research on EU Agriculture; IRR = Internal rate of return; NPV = Net present value; OECD-DAC = Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee; PSM = Propensity score matching; RBM = Results-based management; RCTs = Randomized controlled trials; SNA = Social network analysis; SPIA = CGIAR Standing Panel on Impact Assessment

ACKNOWLEDGEMENTS: The many constructive comments from the following people who reviewed the draft version of the document are gratefully acknowledged: Javier Ekboir (Institutional Learning and Change (ILAC) Initiative, http://www.cgiar-ilac.org/); Tim Kelley and James Stevenson (both from the CGIAR Independent Science and Partnership Council (ISPC) Secretariat, http://impact.cgiar.org); Laurens Klerkx (Wageningen University Knowledge, Technology and Innovation Group, http://www.wageningenur.nl/en/Persons/dr.ir.-LWA-Laurens-Klerkx.htm); Syndhia Mathé (French Agricultural Research Centre for International Development (CIRAD), Innovation Joint Unit http://umr-innovation.cirad.fr/accueil); Matthieu Stigler, Sylvain Quiédeville and Dominique Barjolle (all from the Institut de recherche de l'agriculture biologique (FIBL), http://www.fibl.org/), particularly for their inputs regarding qualitative methods; Christian Grovermann and Karin Nichterlein (both from the FAO Research and Extension Unit, http://www.fao.org/nr/research-extension-systems/res-home/en/); and Julien de Meyer (previously of the FAO Research and Extension Unit), also for his intellectual support to IMPRESA throughout all its early stages. The author is responsible for any errors.

Published on 30 April 2014.

Copyright FAO, 2014.

########################################################################

To unsubscribe from the Impact-L list, click the following link:
https://listserv.fao.org/cgi-bin/wa?SUBED1=Impact-L&A=1

ATOM RSS1 RSS2