True Negatives | Circulation: Cardiovascular Quality and Outcomes

Publishing research with negative results—that is, null or inconclusive findings—is a critical but often overlooked task of biomedical journals. Without it, the scientific literature relies on highly selected pieces of evidence that viewed in isolation can distort a field. Publishing research with negative results provides a more comprehensive and fuller view of a scientific discipline. It also can mitigate wasted time and resources by other scientists who attempt to duplicate research efforts in vain. By filling in the background, it expedites processes for new discovery and can enhance “positive” research by informing the design of future studies.

Yet despite its well-recognized and essential necessities for science, significant challenges exist for authors who seek to publish research with negative results. The label itself invokes an undesirable connotation, often dismissing a negative result as a failure or negligible contribution regardless of whether rigorous study design, execution, and analysis occurred. As a result, such work is frequently less cited, and some have even argued that publishing it can undermine early careers.^1,2 This can make it unattractive for authors to even take the time to write up and submit results when the findings are null or inconclusive. We have become so accustomed to celebrating successes in science that we forget the crucial role of “failures” in advancing a field. Research with negative results (including inconclusive results) has come to be viewed so poorly that even when well-done, antiscience advocates may use it to undermine faith in the entire scientific process.³⁴ The contribution of these data to the research field, however, is critical—its absence indicates publication bias and undermines the whole purpose of systematic review and meta-analyses.

A few such concerns are playing out recently in the example of hydroxycholoroquine as a treatment for the novel coronavirus disease 2019 (COVID-19).⁵ Numerous randomized clinical trials with consistent negative findings have now been published given the topic’s current importance. However, these trials are competing for attention with less well-designed observational studies with outlier and potentially overhyped results. These studies continue to raise lingering questions—some of which are legitimate—on whether the treatment works and may distract from other promising areas of investigation. Indeed, as researchers the world over address the COVID-19 pandemic, the importance of publishing research with negative results has become clearer as we chase a growing list of novel associations that may have little clinical relevance. Be it basic or clinical, the effort should be to develop and execute rigorous research projects and not just report results that are significant. To move forward, knowing what does not work is just as important as knowing what does, especially when resources are limited.⁶

Admittedly, biomedical journals have a history of diminishing the importance of research with negative results. Most journals explicitly take judgments of novelty and anticipated impact into consideration when selecting manuscripts for publication. This is understandable. For example, associations between most features of any system are unlikely to be significant, which is why discovering new relationships is so exciting. And when a null relationship is reported, it is important to identify whether the absence of an association is a scientific truth or simply a study design flaw (eg, an underpowered experiment). Understanding the difference between research with negative results and unimportant research, therefore, requires careful judgment. But systematically overlooking such research is risky. Already a rare spotting in scientific journals, research with negative results is disappearing from the published literature. In a seminal study by Fanelli⁷ from 2011, the rate of research with positive results in >4600 articles across the published literature grew annually by 22% from 1990 to 2007—a finding that was remarkably consistent across disciplines (including clinical medicine). By the end of the study period, fewer than 15% of published articles were classified as having negative results.

This clearly needs to change if we want a more comprehensive and honest view of research; however, biomedical journals have rarely offered a clear pathway toward such an advancement. At the American Heart Association, we are interested in changing this culture and are delighted to report our recent efforts to address this issue. The American Heart Association Journals and Wolters Kluwer have entered into an agreement with the Center for Biomedical Research Transparency (CBMRT, http://www.cbmrt.org/) to create an online Null Hypothesis Collection highlighting important research with negative results that includes inconclusive and confirmatory research articles as well. Our goal is to highlight such articles published by the portfolio to raise awareness of their importance to the scientific community. CBMRT has agreed to provide financial support to make the articles freely available online and will help promote the initiative through their channels. Together we can address publication bias by encouraging the write up and publication of all well-performed studies, including those with negative/null or inconclusive findings. Doing so not only improves research culture and enhances research efficiency but reduces the risk of avoidable harm to patients by ensuring that clinical practice is informed by a more complete and balanced record.

We hope the Null Hypothesis Collection will be an important vehicle for promoting more balance in our publications. For too long, science has been an unfinished portrait with missing details in its background. The subject—the positive discovery—sits flat on the canvas, lacking depth and context. Without crucial details filled in from true negatives in research, it is impossible to understand the full extent of our progress. The Null Hypothesis Collection serves as a tool for showing our conviction to the process of filling in the details and will hopefully encourage authors to continue to submit their rigorous, interesting, innovative research to the American Heart Association portfolio of journals regardless of their reported outcome. Through it, we explicitly recognize that high-quality research is ultimately about more than a new and exciting finding but a product of rigorous study design, execution, and analysis.

Footnotes