#Scientists introduce rating system to assess quality of evidence for policy

The COVID-19 pandemic has underscored the critical need for robust scientific evidence to support policy decisions, such as around the effectiveness of various social distancing measures and the safety of drug therapies. Yet this need arises at a time of growing misinformation and poorly vetted facts repeated by influential sources. To address this gap, a group of scientists led by Kai Ruggeri, a professor at Columbia University Mailman School of Public Health, and James Green, chief scientist at NASA, has introduced a new framework to help set standards for the quality of evidence used in policymaking.

Outlined in Nature Behavioral & Social Sciences, the Theoretical, Empirical, Applicable, and Replicable Impact (THEARI) system ranks evidence in five tiers: (1) theoretical (argument or possible explanation stated), (2) empirical (concept described but not utilized), (3) applicable (concept has been used to elicit effect), (4) replicable (effect has been repeated independently), (5) impact (effect has been appropriately replicated in practice with measurable value in real world). Unlike other evidence ranking systems used in medicine or technology, THEARI applies broadly across disciplines.

The authors say THEARI will help manage risks while also providing a reasonable pathway for applying breakthroughs in treatments and policy solutions in an attempt to stem the harm already impacting the well-being of populations around the world. They hope the system will be applied at two levels: (1) a post-publication as a badge similar to Open Access, and (2) within policy guidance materials. As an example, a cover note to policymakers might say, “we recommend using x approach, which currently has a three-star rating in THEARI. That means it may be useful, but more testing is necessary.”
When Evidence Misaligns with Policy
The authors cite two examples from the COVID-19 pandemic to illustrate the need for an evidence rating system like THEARI. First, they point to a 2007 paper by Cheng et al that warned of a re-emergence of SARS-like coronaviruses; yet despite providing ample high-quality evidence, the warning was unheeded. Second, note the United Kingdom’s decision to delay social distancing measures due to fears of “behavioral fatigue” spreading throughout the population—despite the fact that there was little quality evidence to support the concept.
In the case of hydroxychloroquine, the controversial potential therapy for COVID-19, the authors say there is a lot of evidence about the drug as a therapy for other illnesses, little specific to COVID. Any rating system can only work if applied to the specific context, they note. While the two problematic journal articles on hydroxychloroquine were retracted, policy shouldn’t rely on one or two studies unless genuinely life-or-death in the moment. And any rating system will work if journals have protocols in place to prevent the publication of faulty research.
Beyond COVID-19, the authors observe that climate change, which firmly established as fact in the scientific community, has until recently been presented by the mainstream media as a debate—contributing to delays in policies to address the problem. Opinions presented as theory only receive the lowest THEARI rating; opinions without a theory receive no rating whatsoever. This simplifies the process of understanding what has been studied (sometimes in extreme depth) versus what is simply a perspective, informed or not.
The authors outline several reasons beyond these sorts of mismatches between evidence and policy, including the growing volume of scientific evidence and complexity of political processes, the rapid diffusion of information and misinformation, and the high degree of uncertainties around the reliability and comparability of data.
“Behavioral science suggests that the policy interpretation of existing information can be particularly prone to biases in this context of scarcity of time and resources,” the authors write. “Formulating evidence-informed policies appears to be most challenging right when we most need it.”
The authors conclude: “In presenting THEARI, the ultimate benefit we envision is setting a common framework as a starting point for utilizing evidence in policy discussions, overcoming biases and the effects of inconsistent definitions or unreliable insights. This encourages policymakers to place more value on evidence by providing support for meaningful arguments that may otherwise be disregarded as incongruent with current thinking, even amongst scientists.”