Reading COVID-19 news: What are preprints and why should you care?

As the COVID-19 pandemic continues, every few days we’re seeing reports on alarming or exciting new developments – new tests, new treatments, scary new mutations. How is a reader to figure out what is real and what should be treated with skepticism? One thing to check is whether the story is based on one of the many preprints being posted each day versus research that’s been through peer-review. But what is a preprint and how can you tell that one is the source?

What’s a Preprint

In the usual course of doing medical research, studies and experiments are submitted to professional journals where they undergo peer-review — a process where ideally unbiased researchers with expertise in that area of research review the paper thoroughly, point out flaws that need to be addressed, and recommend that the journal’s editors either accept the paper, reject it, or request that the authors revise in response to the reviewers’ comments, and resubmit the edited version. While there are problems with this process and flawed papers can end up in even the top journals, it provides a level of quality control. It also takes a long time — many months, or even years.

Over the past few decades, fields such as physics and math have created what are called preprint servers — online repositories where researchers can post draft papers, many of which are later submitted to journals. Anyone can read them and, if they wish, post comments critiquing them. The positive is that research can be shared quickly. Unfortunately, this means misinformation can also be shared quickly.

The Good

Until recently, this was not common in biomedical research. However, in the midst of a pandemic, speed becomes crucial. In the past few months, publishing preprints of COVID-19 research has become commonplace as a way for researchers to share information with each other. While the preprints haven’t undergone peer-review, readers can post comments pointing out errors or gaps, then have discussions with each other and the researchers. In addition, there are active research communities on Twitter such as #epitwitter where new papers are dissected in detail.

The Bad

The main two preprint repositories in the health sciences are bioRxiv and medRxiv, which are now linking jointly to COVID-19 research. Each has a notice on its home page stating:

A reminder: these are preliminary reports that have not been peer-reviewed. They should not be regarded as conclusive, guide clinical practice/health-related behavior, or be reported in news media as established information.

Unfortunately, those notices are about as effective as speed limit signs. Journalists are under pressure to get news out quickly, especially exciting news and the flashier the research finding, the more clicks and shares articles about it will get.

What’s a Reader to Do?

When you see the latest news headlines about a new amazing cure or a wonderful vaccine or how a mutant coronavirus strain is spreading:

• Always be skeptical of dramatic results. The more exciting or frightening the news, the more carefully you should check the article.
• Check where the study is published. If the source is bioRxiv or medRxiv, remember that anyone can post a paper there and no one has reviewed the paper or checked the results.
• What is the researcher’s area of expertise? We’re seeing a lot of non-infectious disease epidemiologists modeling projections of things like the spread of COVID-19, how hospital capacity is likely to hold up, how well can flattening the curve work without understanding what factors are important to include in the models and how they interact. Anyone with a strong math background can make a model, but it takes education in the epidemiology of infectious disease to make a good model.
• Has the journalist interviewed experts in this particular area who have had time to thoroughly examine the paper? This can mean experts on coronaviruses, infectious disease epidemiology, or other specialized areas.
• If the research is a clinical trial, is it well designed? Optimally, the reporter will have an expert in clinical trials review it but if they don’t, some things to look for are:
  • was there was a comparison treatment (if not, you can’t tell if the patients would have improved anyway),
  • were participants randomly assigned to study arms (otherwise you can end up with all the sicker patients or those with a certain risk factor or in a certain age group or…) on one arm, making it impossible to tell if treatment differences are real or due to these imbalances
  • were there enough participants to be able to draw conclusions*
  • was any difference found clinically meaningful
  • remember that not finding a “statistically significant” difference doesn’t mean there’s definitely no difference, just that we can’t tell yet (absence of evidence doesn’t equal evidence of absence)
  • did one treatment cause more harm (for example heart attacks, infections, liver damage)

In addition to the above, see if the story lasts. Are there follow-up articles confirming or refuting the news? Is it a flash in the pan that disappears after a couple of days? Does it subsequently show up in a peer-reviewed journal (many of which are speeding the review process for COVID-19 papers)?

Above all, remember that science is a process of trying to increase and correct our knowledge. We should expect that some of what we heard at the start of the pandemic turned out to be wrong, and some of what we think we know today will be corrected or refined in the future.

* While “large enough” varies by type of study and the size of the effect found, in general you’d like to see at least 40 or more participants in a preliminary study and several hundred in a Phase III clinical trial. If the sample size is small enough that changing the results for 2 participants has a major effect on the findings, it’s way too small.