Michael Kevane

For my African Economic Development class we are reading a paper that uses the randomized control trial methodology to evaluate a program to encourage farmers to produce crops for export.  The presumption is that new crops like french beans, passion fruit, and baby corn could be sources of higher incomes for farm households.  The paper is “Finding Missing Markets (and a disturbing epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya” by Nava Ashraf, Xavier Giné, Dean Karlan and appeared in American Journal of Agricultural Economics in 2009.

The RCT involved randomly allocating “self-help” groups of farmers into three categories: control, those receiving extension services about export crops, and those receiving extension and also credit.  The program was offered by a Kenyan outfit known as DrumNet (for profit? non-profit? unclear to me!).  There were 12 groups in each arm, and each group consisted of about 30 farmers, so there were about 370 farmers in each treatment arm and control group.  Overall, then, a total of about 1,200 farmers were involved.  If, however, the farmers in the self-help groups were very homogeneous (their characteristics are closely correlated), then the sample size was in effect just 12 in each arm.  So that clustering of the treatment into possibly very homogeneous farmer groups has to be taken into account in the analysis.

Interestingly, and not explained in the paper, is that fewer than half of the self-help groups and only 27% of the farmers selected chose to actually participate in the treatment arm that was just “expert advice” and marketing assistance.  That is a low take-up rate.  It seemed that middle income farmers (and not low or very high income farmers) were more likely to participate in the program.  But the researchers apparently did not actually ask farmers why they did not avail themselves of the project.

To evaluate the impact of the program, the researchers conduct a regression difference-in-difference analysis using baseline values of certain variables and seeing how the outcomes were different after a year, according to which treatment the farmers received.  The researchers estimate what is sometimes called the “intent to treat” effect, and seemed not to incorporate the knowledge that many farmers refused to participate into the analysis (by estimating an average treatment effect using the treatment assignment as an instrumental variable for the actual participation variable).  The researchers also do no adjustment for the fact that they are examining the intent to treat effect on ten different outcome variables.  Usually one might set a tighter standard for concluding that an effect is statistically significant at conventional levels.

In any case, here is the researchers summary conclusion from the regression analysis: “We find that the program succeeds in getting farmers to switch crops, and that the middle income farmers were the most likely to take-up (relative to low-income and high-income). Comparing members that were offered credit to those that were not, we find that credit increases participation in DrumNet but does not translate into higher income gains relative to the non-credit treatment group. This suggests that access to credit is not necessarily the primary explanation for why farmers are not accessing these markets on their own. We find a significant increase in household income but only for farmers who were not previously accessing export markets.” [emphasis added]

I do not like that last heterogeneous treatment effect.  After they find no effect on household income in the main regression analysis, they decide to look for interaction effects.  How does the reader know they did not try a dozen different interaction effects, and have only reported the one that was significant?

The paper could prompt a lot of hypotheses and interesting research projects.  For example, at one point the authors mention that the farmers generally rejected the passion fruit technology. The only reason offered is that passion fruit is “challenging” even though “profitable.”  Gee, isn’t everything that is profitable challenging?  It is only the lucky entrepreneur who says, “Making money is easy!”  But actually when you read the passion fruit success stories, like this one from Nzaui and this one from Murang’a, it does sound easy.  Another example is that the entire export promotion effort by DrumNet (a spinoff of something called Africa Pride), apparently collapsed and farmers lost a lot of money.  The authors allude to this in their “epilogue” but offer few details.  None of them seemed to have that investigative reporter instinct to follow a story.  The more I web-search DrumNet and Africa Pride, the more suspicious I get of the organization.  It would be nice for someone at the World Bank and the Gates Foundation to clarify just how much concessional funding (grants? loans?) they channeled to the organization.

Incidentally, a replication study by 3iE using the original data found the original analysis to be basically replicable, though the replicated estimated impacts were smaller than the original.  But the replication study authors did note that the study was very under-powered (the sample size was much too low): “In terms of the statistically insignificant increases to  household income, our analysis suggests future evaluation would need to substantially increase sample sizes, on the order of quadrupling the original sample size, to be able to detect a statistically significant difference  between treatment and control groups in this regard.”

I am teaching African Economic Development this quarter, and I like to have students watch films and read novels to get a more humanistic understanding of the people behind the numbers.  If you have never spent any time in any African country, or only spent time in African cities, for that matter, it is easy to “other” rural folk.  So I assigned a nice documentary called Duka’s Dilemma, by Jean Lydall and Kaira Strecker.  What I like about the documentary is it focuses on just four people, and after 80 minutes you feel like you can recognize a little bit of their personalities, and the interpersonal relations among them.  They are no longer other.

What I didn’t realize is that the Hamar occupy a kind of voyeuristic status that precisely contributes to the othering.  And they seem to be comfortable with selling their othering to “primitive people” photographers.  Sigh.  makes me wish I were in the humanities and could spend all day talking and thinking about those kinds of issues.

This is a decent, unnarrated, video showing what a typical market day is like in the “town” presumably like the one mentioned in Duka’s Dilemma.

As always, once you get to know people, you start doing a little digging, and you are more sympathetic to how big, forces (but always individuals making choices) are changing their world.  From Survival International:

A massive hydroelectric dam and associated land grabs for plantations threaten the tribes of the Lower Omo River.  The tribes have lived in this area for centuries and have developed techniques to survive in a challenging environment. They have not given their free, prior and informed consent for the dam or the plantations and have already started to lose their livelihoods based on the river’s natural flood cycle.

As more and more development economists conduct randomized control trials (RCT), rather than inferring the answer to an economics question from observational data, some researchers are drawing attention to unmentioned assumptions underlying interpretations of RCT results.  One such implicit assumption concerns participant response to the expectation of treatment, rather than response to the actual treatment. The response is relevant for “blind” RCTs where the treatment and control groups do not know which group they are in.  The expectation of a probability of receiving the treatment may change behavior, and this leads to a different outcome (a different difference between the treatment and control group) compared with an experiment that is not blind (so the control group knows it is not getting the treatment).  Likewise, there might be differential estimates of the treatment effect in experiments where the probability of receiving the treatment is higher or lower.  Chassang et al in a paper  “Accounting for Behavior in Treatment Effects: New Applications for Blind Trials” show that randomizing the probability of treatment assignment in a blind trial (and letting participants know the different probabilities) can then enable inference of the importance of the effect of behavioral response interacted with treatment.

A paper I have assigned for my African Economic Development class made the headlines (OK maybe not headlines but a couple of blog and Twitter commentaries) back in 2012 and 2014 for using the Chassang et al result to measure and use this kind of respondent behavior in a typical development and agricultural economics setting.    The paper is “Behavioural Responses and the Impact of New Agricultural Technologies: Evidence from a Double-Blind Field Experiment in Tanzania” by Erwin Bulte, Gonne Beekman, Salvatore Di Falco, Joseph Hella, and Pan Lei, in AJAE 2014.

Bulte et al run two experiments: one where farmers know which variety of cowpea seed they are given (traditional or improved) and one where farmers do not know (they are “blind” but can infer that the probability of getting the new improved variety is about 50%).  In the first experiment, where farmers know the variety, the improved variety has about 25% more  yield (9.87 vs 7.24).  In the blind experiment, the two seeds have about the same yield (9.91 vs 9.40).  Notice that in the blind experiment the yield is about the same as in the open experiment where farmers use the improved variety (all about 9.5).  Bulte et al interpret this to suggest that all of the increase in yield in the open trial is likely due to changed cultivation methods (better weeding, spacing, planting, thinning, pest control, etc.) and very little to the seed itself.  What seems to have happened is that when farmers knew they had the traditional variety, they farmed it in a way that generated lower yield (and note that lower yield may have been the profitable way to farm cowpea; Bulte et al can say very little about the costs of cultivation practices that increased yield).

This result is very reasonable.  It suggests that the effects estimated by open RCTs are likely a combination of behavioral and mechanical (i.e. laboratory) effects.  One development wag suggested the paper was “a torpedo aimed straight at H.M.S. Randomista.”  But I think their paper is more constructive than destructive.  What Bulte et al show is that running two trials (an open and a blind trial) enables a researcher to make an inference about how much of an effect is behavioral and how much is mechanical.  In their case, the results suggest that most of the new seed effect is behavioral.  (Berk Ozler pointed out in comments on the 2012 draft that the Bulte et al paper has lots of other problems, but here I set all those aside.)

Many economists will probably respond with something like, “But none of our interventions could ever be done blind; the subject always knows whether they are in treatment or control.”  But the more you think about it, the more examples you can think of where that is not true.  Two come to mind.  Any parent of schoolkids knows that school testing relies precisely on this not knowing.  The teachers strongly suggest to students that school tests are high stakes, and matter.  Parents obfuscate when their middle schooler asks.  So many students think they matter.  Once students mature and start to figure out and share with others that tests do not matter, performance changes.  The perceived probability of being in a high stakes experiment affects the outcome.  So you are measuring an interaction of knowledge and belief that demonstrating knowledge matters.  The same goes for many experimental games.  One could imagine an experiment designed to measure competitiveness (by placing respondents into absolute performance pay or relative performance pay arms) vary the probability of being in one of the two arms.  Then the behavioral response (“Oh I am in the competitive pay situation I am supposed to behave this way”) can be perhaps separated from the (semi-unconscious?) psychological response.