I have a forthcoming paper (with Dan Hicks and Weici Yuan) that has a running head of “Peacocks in Porsches”. I once tried to publish a paper called “How Dead is the Solow Model” (the abstract was two words: “Stone dead”).
So you can imagine how impressed I was to see a piece in the new AEJ: Applied called “Star Wars: The Empirics Strike Back”.
An ungated version of the paper is available here.
Star Wars in this context refers to the barbaric practice of putting “stars” beside coefficients in regression tables that are significant. The lower the p-value, the greater the number of stars.
The authors investigate the distribution of p-values across tens of thousands of coefficients in published economics articles and find,
“The distribution of p-values exhibits a camel shape with abundant p-values above 0.25, a valley between 0.25 and 0.10 and a bump slightly below 0.05. The missing tests (with p-values between 0.25 and 0.10) can be retrieved just after the 0.05 threshold”
In other words, if your p-value is worse than .25, most researchers will not try to “rescue” their test. But if you get close to the holy grail of at least marginal significance, it appears that many researchers will not report that test and work to find a specification that pushes the test into “significance”.
It’s all about them stars.