For my African Economic Development class we are reading a paper that uses the randomized control trial methodology to evaluate a program to encourage farmers to produce crops for export.  The presumption is that new crops like french beans, passion fruit, and baby corn could be sources of higher incomes for farm households.  The paper is “Finding Missing Markets (and a disturbing epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya” by Nava Ashraf, Xavier Giné, Dean Karlan and appeared in American Journal of Agricultural Economics in 2009.

The RCT involved randomly allocating “self-help” groups of farmers into three categories: control, those receiving extension services about export crops, and those receiving extension and also credit.  The program was offered by a Kenyan outfit known as DrumNet (for profit? non-profit? unclear to me!).  There were 12 groups in each arm, and each group consisted of about 30 farmers, so there were about 370 farmers in each treatment arm and control group.  Overall, then, a total of about 1,200 farmers were involved.  If, however, the farmers in the self-help groups were very homogeneous (their characteristics are closely correlated), then the sample size was in effect just 12 in each arm.  So that clustering of the treatment into possibly very homogeneous farmer groups has to be taken into account in the analysis.

Interestingly, and not explained in the paper, is that fewer than half of the self-help groups and only 27% of the farmers selected chose to actually participate in the treatment arm that was just “expert advice” and marketing assistance.  That is a low take-up rate.  It seemed that middle income farmers (and not low or very high income farmers) were more likely to participate in the program.  But the researchers apparently did not actually ask farmers why they did not avail themselves of the project.

To evaluate the impact of the program, the researchers conduct a regression difference-in-difference analysis using baseline values of certain variables and seeing how the outcomes were different after a year, according to which treatment the farmers received.  The researchers estimate what is sometimes called the “intent to treat” effect, and seemed not to incorporate the knowledge that many farmers refused to participate into the analysis (by estimating an average treatment effect using the treatment assignment as an instrumental variable for the actual participation variable).  The researchers also do no adjustment for the fact that they are examining the intent to treat effect on ten different outcome variables.  Usually one might set a tighter standard for concluding that an effect is statistically significant at conventional levels.

In any case, here is the researchers summary conclusion from the regression analysis: “We find that the program succeeds in getting farmers to switch crops, and that the middle income farmers were the most likely to take-up (relative to low-income and high-income). Comparing members that were offered credit to those that were not, we find that credit increases participation in DrumNet but does not translate into higher income gains relative to the non-credit treatment group. This suggests that access to credit is not necessarily the primary explanation for why farmers are not accessing these markets on their own. We find a significant increase in household income but only for farmers who were not previously accessing export markets.” [emphasis added]

I do not like that last heterogeneous treatment effect.  After they find no effect on household income in the main regression analysis, they decide to look for interaction effects.  How does the reader know they did not try a dozen different interaction effects, and have only reported the one that was significant?

The paper could prompt a lot of hypotheses and interesting research projects.  For example, at one point the authors mention that the farmers generally rejected the passion fruit technology. The only reason offered is that passion fruit is “challenging” even though “profitable.”  Gee, isn’t everything that is profitable challenging?  It is only the lucky entrepreneur who says, “Making money is easy!”  But actually when you read the passion fruit success stories, like this one from Nzaui and this one from Murang’a, it does sound easy.  Another example is that the entire export promotion effort by DrumNet (a spinoff of something called Africa Pride), apparently collapsed and farmers lost a lot of money.  The authors allude to this in their “epilogue” but offer few details.  None of them seemed to have that investigative reporter instinct to follow a story.  The more I web-search DrumNet and Africa Pride, the more suspicious I get of the organization.  It would be nice for someone at the World Bank and the Gates Foundation to clarify just how much concessional funding (grants? loans?) they channeled to the organization.

Incidentally, a replication study by 3iE using the original data found the original analysis to be basically replicable, though the replicated estimated impacts were smaller than the original.  But the replication study authors did note that the study was very under-powered (the sample size was much too low): “In terms of the statistically insignificant increases to  household income, our analysis suggests future evaluation would need to substantially increase sample sizes, on the order of quadrupling the original sample size, to be able to detect a statistically significant difference  between treatment and control groups in this regard.”