This is Matthieu Stigler again (my first message was nr. 15).
Coming back to Message 20 by Peter Midmore, I think there is some confusion that has slipped in between two similar, yet distinct concepts: self-selection (Message 13, by Amadou Binta Ba) and sample selection (Message 20). While similar, these concepts have implications at different levels, threatening either "internal" or "external" validity. Quickly defined, internal validity is whether conclusions of a study validly apply to the sample considered, while external validity is whether conclusions that are valid for a sample also apply to the larger population, or other populations not considered - in other words, whether the results can be generalised to other contexts.
So for Amadou's point, when you have self-selection, this means that within the population you considered, participants might differ from non-participants, implying that you might not be able to find a relevant control group. This means you will not obtain internal validity, as you are not able to describe well the effect of the programme.
On the other side, sample selection means you are not able to observe the whole population, and base your conclusions on a part of the population. This can be within a quantitative study, where you miss some individuals (which differ from the other in a non-random way), or within a qualitative case study, where the simple fact of choosing a case means you do not observe all the cases. With such sample selection, internal validity might still be preserved (your conclusions validly apply for the sample you considered), but external validity is not guaranteed. So obviously, such sample selection is an issue only if you want to be able to claim that your results apply to other contexts (regions, product, etc).
Now Amadou, you mention the presence of "spill-overs effects", which is yet another point. This is considered bad when the "control" participants also adopt the technology, and hence cannot be used anymore as control. If other villages outside your sample also adopted the technology, this should not be an issue for your sample, and could make your claim actually stronger, in the sense that the benefits of the programme could be even higher than what you estimate. [The reference here seems to be Amadou's second message, nr. 24, where she mentioned her concerns about the fact that "During an interview with local community leaders of my study area I have been told that the project had an effect beyond the target villages. It impacted some neighboring villages because of their position. I think that failure to take into account this type of selection bias can affect the results."...Moderator].
Now turning to Peter Midmore's initial point (Message 20), advocating the use of failure case to balance selection of too many positive cases, and Ed Garrett's reply (Message 29), suggesting selecting cases at random, I wanted to mention (although I do not entirely share it) a quite different position in the CGIAR report by Walker et al. (2008), pages 23-26. Remarking that in most cases, agricultural R & D leads to no results, but that in some cases it leads to huge results (hence projects's benefits have a skewed distribution), Walker and colleagues criticise the random selection idea, arguing that it would give a wrong picture, giving almost no chance for success cases to be selected. Going further, they suggest that one just needs to focus on "success stories", which help best understand mechanisms to success.
Matthieu Stigler
Institut de recherche de lšagriculture biologique (FiBL) Ackerstrasse 113, Case postale 219
5070 Frick,
Switzerland
www.fibl.org
e-mail: matthieu.stigler (at) gmail.com
Reference:
- Walker, T., Maredia, M., Kelley, T., La Rovere, R., Templeton, D., Thiele, G. and B. Douthwaite. 2008. Strategic guidance for ex post impact assessment of agricultural research. CGIAR Independent Science and Partnership Council (ISPC). http://www.sciencecouncil.cgiar.org/fileadmin/user_upload/sciencecouncil/Impact_Assessment/SC_epIA_low-res_for_web.pdf (0.5 MB)
[To contribute to this conference, send your message to [log in to unmask] For further information, see http://www.fao.org/nr/research-extension-systems/res-home/news/detail/en/c/217706/ ].
########################################################################
To unsubscribe from the Impact-L list, click the following link:
https://listserv.fao.org/cgi-bin/wa?SUBED1=Impact-L&A=1
|