“Time Series Minimum Wage Literature: A Meta-Analysis”
One way publication bias can have impacts beyond the scientific community is when poor policy recommendations are made based on skewed results. Such was the case when time-series minimum wage studies were used to make recommendations against raising the minimum wage in the United States. This video introduces David Card and Alan Krueger’s meta-analysis of these studies and demonstrates the effect of increasing an experiment’s sample size on the precision of its results.
In the article, economists David Card and Alan Krueger analyze the effect of an increase in the minimum wage on unemployment using aggregated time-series studies. Inspired by the widely-believed prediction that an increase in the minimum wage will lower the employment rates of low-wage workers, they estimated the probability of publication bias in studies on the relationship between changes the two variables.
Because more recent studies have found either smaller effects or marginally positive effects of the minimum wage on employment levels, and because of the role time-series evidence plays in minimum wage literature, Card and Krueger look to determine the validity of these claims.
They present a meta-analysis of published literature, building on the observation that “more recent studies have access to many more observations than earlier studies.” Card and Krueger define meta-analysisas “the quantitative analysis of a body of studies” that can be used to “summarize a set of related studies,” “evaluate the reliability of the findings in a statistical literature,” and “test for publication bias.”
They state that “basic sampling theory suggests that there should be a simple ‘inverse-square-root’ relationship between the sample size and the t ratio obtained in different studies.” However, their “findings are difficult to reconcile with the hypothesis that the literature contains an unbiased sample of the coefficients and t ratios that would be expected given the sample sizes used in the different studies.”
After finding t ratios that are actually negatively correlated with sample sizes, they conclude that “the time-series literature may have been affected by a combination of specification searching and publication bias, leading to a tendency for statistically significant results to be overrepresented in the published literature.” Note: specification searchingis the technical term for what is widely known less formally as p-hacking.
Publication bias comes into play when authors are aware of reviewers’ tendency to give more credibility to studies with statistically significant results. Thus, economists who believe that a rise in minimum wage will lower employment may design their analyses (i.e., choose their variables, select their samples, specify their techniques, etc.) to generate the desired negative and significant effects.
In conclusion and considering reasons why published t ratios tend to equal 2 – the threshold significance value – regardless of the magnitude of the minimum-wage effect, Card and Krueger suggest two possible explanations:
- Structural change – While the true effects of changes in the minimum wage may have departed from earlier predictions, there is little incentive to report such changes because they challenge the validity of the existing time-series approach.
- Specification-searching and publication bias is a more plausible explanation, however – Because of the predominant theory, authors and editors tend to look for negative and statistically significant effects and will try to replicate and reproduce these results.
Due to the high probability of publication bias and specification searching, it is probable that “‘insignificant’ or ‘wrong-signed’ results may be substantially underreported in the published literature.”
You can read the full article here.
Reference
Card, David, and Alan B. Krueger. 1995. “Time-Series Minimum-Wage Studies: A Meta-Analysis.” The American Economic Review 85 (2): 238–43.
© Center for Effective Global Action.