What we can (and can’t) take away from New York’s antibody testing results

New York Gov. Andrew Cuomo on Thursday offered what appeared to be a surprising statistic on just how widespread the coronavirus outbreak has hit the state: Almost 14 percent of people tested in a new study were found to have COVID-19 antibodies.

It might be good news. The data suggests the state’s mortality rate could be lower than previously thought, albeit much higher than seasonal influenza. But it’s still too early to reach that conclusion — or any other extrapolations people might want to make about easing lockdowns, experts say.

Beyond that data point, Cuomo’s presentation offered little detail that would allow a full assessment of whether that many New Yorkers were really infected. The numbers were from a preliminary analysis of a study that was not released publicly. As such, most of its methodology remains unknown. The New York State Department of Health did not respond to a request for a copy of the study.

That doesn’t mean the findings are wrong. But it does mean that extreme conclusions about the mortality rate or the state’s ability to reopen shouldn’t be extrapolated from the data.

“There’s a risk of really serious misinterpretation here,” said William Hanage, an associate professor of epidemiology at Harvard’s T.H. Chan School of Public Health in Boston. “But the most basic conclusion — that quite a large number of people may have been infected and are not turning up in the official case counts — that’s extremely plausible and something we have been suspecting all along.”

The confusion highlights just how little is still known about the coronavirus as well as the difficulties of doing difficult scientific work under the lens of a global microscope. Researchers are still piecing together important details about the virus, including just how fatal it is.

Understanding just how many people have been infected by the coronavirus is critical for public health officials to gauge when it might be safe for states to begin easing strict social-distancing measures. Together with expanded diagnostic testing and contact tracing, the results of antibody tests can help states contain the spread of the virus and prevent local health care systems from being overwhelmed with subsequent waves of infections.

That has made stats like the one offered up by Cuomo into a crucial piece of news — but also one prone to confusion and miscalculation. Experts caution that preliminary findings from this study, and similar research that was released in California in the past week, could be skewed by statistical issues, unreliable test results and questionable ways that the studies themselves were designed.

The data is particularly crucial as a national debate has emerged over when and how to begin to ease lockdowns, with some conservative pundits pointing to prevalence tests like these as proof that the U.S. has overreacted to the outbreak. Public health professionals have been nearly unanimous in their disagreement with those assessments.

The New York study found that out of 3,000 grocery store customers who were randomly tested across the state, 13.9 percent were positive for COVID-19 antibodies, indicating they had an active infection or had been infected and recovered. In New York City, 21.2 percent of those tested were positive for COVID-19 antibodies, according to Cuomo, who presented the preliminary results of the state’s study Thursday in a news briefing.

If those infection rates are correct, it could mean that more than 2.7 million people in New York — and roughly 1.8 million people in New York City alone — have been infected, far more than the state’s 257,216 confirmed cases.

Though these early results suggest that infections in New York have been much more widespread than the official counts indicate, experts say getting a clear picture of the true number of coronavirus cases is extremely complex, and it’s still not known whether, or how, COVID-19 antibodies translate into immunity from the virus.

“Outbreaks are like an iceberg, where the top part that we can see are the people who are hospitalized or die, but there’s a big bottom part made up of people who were missed,” said Natalie Dean, an assistant professor of biostatistics at the University of Florida in Gainesville. “Establishing how big that bottom part is helps us characterize the severity of the disease.”

But some have called into question the accuracy of some antibody tests and how these early studies have accounted for the possibility of false-positive results.

The tests used in New York were developed by the Wadsworth Center, a public health laboratory of the New York State Department of Health. In a fact sheet provided by the agency, the test’s specificity — as in its ability to identify antibodies specific to this coronavirus rather than previous coronavirus outbreaks — is between 93 percent and 100 percent. The test’s sensitivity, or its ability to deliver accurate positive results, was not listed.

Carl Bergstrom, a professor of biology at the University of Washington, said the early results of the New York study seem reasonable, particularly given the number of deaths recorded in New York City. He added that the plausibility of the findings are bolstered because the testing occurred in a particularly hard-hit region of the country. As a result, the probability that a positive test truly reflects that someone has the disease rather than being a false positive is higher.

“If you’re using an error-prone test in a low-incidence area, you’re not getting much information,” Bergstrom said. “But if you’re testing a population with 20 percent infected, like in New York City, the chance that any particular positive is a true positive is much higher.”

But Florian Krammer, a professor of vaccinology at the Icahn School of Medicine at Mount Sinai in Manhattan, questioned the infection rate for New York City, saying it seemed too high.

“I would think 6-8%, maybe 10% are closer to the truth. It would be nice to know more about the test, its sensitivity and specificity and the test population,” he tweeted in response to Cuomo’s presentation.

Accuracy concerns similarly plagued a pair of California-based antibody surveys, both of which have yet to undergo the standard peer-review process that is an essential to any scientific research. One of the studies, led by researchers at Stanford University, suggested that coronavirus infections in Northern California could be up to 85 times higher than official counts. And another study, led by researchers at the University of Southern California, suggested that the true number of infections in Los Angeles County could be up to 55 times higher than what has been reported.

The two California studies ignited fierce debates within the scientific community, with some scientists claiming that the authors did not adequately account for the rate of false positives with their tests, which likely influenced the findings.

With the Stanford survey, some scientists also criticized the study’s methodology, claiming that the way people were recruited for testing — through ads on Facebook — introduced biases that likely skewed the results.

Though surveying shoppers at grocery stores may be more randomized than recruiting through social media ads, there are still questions about how much the New York study truly represents the population.

“These are people who were out and about shopping,” Cuomo said in the Thursday briefing. “They were not people who were in their homes, they were not people who were isolated, they were not people who were quarantined.”

Still, antibody surveys are a valuable way for public health officials to grasp the scale of an outbreak. If researchers know the true number of coronavirus cases, for instance, they can also more accurately pin down the disease’s fatality rate.

In New York, the fatality rate based on the number of official cases was more than 7 percent, but that number was long thought to have been inflated because diagnostic testing shortages meant only the most serious cases were being confirmed.

If the infection rates based on the state’s antibody testing holds true, New York’s mortality rate would be closer to 0.5 to 1 percent — still higher than, for example, the estimated 0.1 percent mortality rate in the U.S. from the seasonal flu. However, Cuomo qualified that these calculations were based on the state’s more than 15,500 reported deaths in hospitals and nursing homes, and do not include potential at-home deaths.

While a lower-than-predicted mortality rate is welcome news, there is a danger that the adjusted figures could cause people to underestimate the severity of the disease, according to Hanage.

In New York, for example, even if the mortality rate ends up being lower than 7 percent, the coronavirus has exacted a serious toll.

“The number of deaths in New York is already considerably more than you would expect if every single man, woman and child in the city had been infected with the flu,” Hanage said. “If you then think about how without any countermeasures, it would eventually infect 70 or 80 percent of the population, then hopefully people start to understand that this is still very dangerous.”