Poor Transparency of Legal Methods in Empirical Studies of the Effects of Law: A Case Study of Health Policy Research

Vollständiger Abstract

Worum geht es in dieser Arbeit?

Law is a primary mechanism to promote socially desirable behaviors and standards, and so an important subject of evaluation research. Research credibility depends on meeting scientific standards of transparency and replicability. Prior research suggests that information on methods used to construct legal variables in policy research is often not reported. We assessed the transparency of legal research methods in 100 highly cited health policy impact studies published between 2009 and 2019. Papers and any cited sources of legal data were examined independently by two researchers to capture information provided about legal research, coding, and quality control. Coding was reviewed and discrepancies resolved by an author. Papers were rated as Transparent, Partly Transparent, or Not Transparent. Fifty-eight of one hundred papers in our study sample were Not Transparent. Sixteen papers were Partly Transparent, providing methods information on legal research or coding but not both. Eleven studies provided no information about the source of either their legal research or coding, and six provided no information about both. Most papers obtained legal information from secondary sources, including 40% of studies that relied on websites that were not primary sources of law, not designed to produce scientific data, and not transparent as to how their legal research and analysis were performed. Readily adopted improvements in legal research transparency can facilitate replication, support further methodological progress, uphold basic norms of science, and maintain public confidence in policy research.