Gelman and Loken nail it (for me)…. p-hacking and fishing need to be taken more seriously in Economics

I loved reading this article.

The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time

Andrew Gelman and Eric Loken

Abstract: Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data. The problem is there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of shing or examining multiple p-values. We discuss in the context of several examples of published papers where data-analysis decisions were theoretically-motivated based on previous literature, but where the details of data selection and analysis were not pre-specifed and, as a result, were contingent on data

It isn’t the final word, but it is amazing this conversation, started by Ed Leamer more than 30 years ago, is still going on.

About mkevane

Economist at Santa Clara University and Director of Friends of African Village Libraries.
This entry was posted in Development thinking, Education effects. Bookmark the permalink.