Scientists working with the World Health Organization (WHO) have corrected some startling errors in their estimates of how many deaths the pandemic has caused, following a series of questions on the original WHO report, published in early May.
In a review of a technical paper on their methods, researchers reduced the estimated pandemic-related deaths in Germany by 37%, making their excessive mortality rate lower than in the United Kingdom. Spain1. They also increased their estimate for Sweden by 19% (see “Corrected estimates of pandemic mortality rates”).
The WHO study published on May 5 had estimated the excess mortality rate, that is, the increase in mortality above expected levels, for 194 countries. The organization reported that between 13.3 and 16.6 million people had died worldwide between January 2020 and December 2021 due to the pandemic, more than 2.5 times the death toll from VOCID- 19 informed. The estimate was more conservative than other analyzes of excess deaths (see “The True COVID Toll”).
Sources: Our world in data /The economist/ IHME / QUI
But some observers soon expressed concern about the figures for certain countries, especially Germany. It was thought to have coped better with COVID-19 than many other European countries, but the WHO estimated that its excess mortality rate was higher than that of many of its neighbors.
“Almost immediately, we realized that there was a problem,” said Jon Wakefield, a statistician at the University of Washington in Seattle who leads the WHO global COVID-19 death project and publicly tweeted the document. revised May 18th. He says the team is now in the process of re-examining all of its estimates.
The WHO project is a live model, which researchers said will always be updated as demographers learn more. The organization has not yet changed the numbers on its project website (see go.nature.com/3azupk5). Official systematic updates to excess mortality estimates, including for Germany and Sweden, will continue “in the next iteration scheduled for the end of the year,” says Somnath Chatterji, a senior adviser in the Data, Analysis and WHO Impact Delivery to Geneva. Swiss.
The mistakes are important because the WHO study quickly received media attention around the world as an official estimate of the actual number of lives lost as a result of the pandemic. The project is also politically sensitive: some critics used the first set of incorrect estimates to challenge Germany’s pandemic policy. And the Indian government disputes the WHO’s estimate of 3.3 million to 6.5 million deaths in the country, which is about 10 times the official death toll for COVID-19. India. (Other researchers say the WHO estimate is more reliable than the Indian government’s estimate; the WHO figure is also in line with several other studies).
In an interview with Nature, Wakefield explained the problems his team encountered in his work. “We want to get this out of the way because it’s wrong. We have to fix it,” he said.
The big picture
To determine how many people died as a result of the pandemic, researchers model all deaths during the period and subtract a baseline of expected deaths (those that would have occurred in the absence of a pandemic). What remains is the above-average deaths: a more reliable measure of pandemic-related mortality than official figures provide, because many countries have not reported or not the deaths from COVID-19.
This type of project can only provide approximate approximations, because it requires complex modeling and periodic review as new data arrives. For example, so far only 100 countries in the world have reported national deaths each month for at least part of the pandemic period. , says the WHO. However, WHO figures showed that some countries, such as India, Russia and Egypt, had massively underestimated their deaths from COVID-19: the excess deaths of these countries during 2020-21 was much higher than its official COVID-19 tolls. Estimates also showed countries that had many more deaths than normal levels, several of them in South America. Peru stands out, with an increase in deaths that almost doubled its usual mortality during these two years.
What surprised the critics, however, were some of the results from rich countries that report timely deaths, such as Germany and Sweden. The problem seemed to be in the way the expected deaths had been modeled. A few hours after the publication of the WHO results, commentators on Twitter noted that the organization’s prediction of expected mortality in Germany in 2020-21 was surprisingly low, which increased the number of deaths in excess. .
Where the WHO was wrong
Researchers model expected mortality by extrapolating historical trends. For example, the World Mortality Dataset (WMD), a widely cited project, uses a linear extrapolation of deaths in 2015-19 to account for underlying mortality trends. A researcher on this project, economist Ariel Karlinsky of the Hebrew University of Jerusalem in Israel, was also part of the Wakefield technical team. However, the WHO group used a mathematical function called a thin plate spline to estimate the expected deaths by 2020-21. Unfortunately, Twitter commenters noted, this feature appeared to be too sensitive to a slight drop in German deaths in 2019; he also predicted a decline in deaths in 2020 and 2021 (see ‘The German puzzle’).
Fonts: World Mortality Dataset / WHO / J. Wakefield
“Extrapolating a spline is a well-known bad practice,” says Jonas Schöley, a demographer at the Max Planck Institute for Demographic Research in Rostock, Germany. Nature spoke with other demographers, who agree.
Schöley was asked to consult the WHO technical paper in April and says he warned Wakefield that spline extrapolations could lead to “predictions of erratic trends”. But at the time, the data and estimates had been recorded by the WHO team; Schöley was being consulted for a different aspect of the play.
Following criticism, Wakefield and the WHO team re-examined their extrapolation method. But then they discovered a second problem, which turned out to be a bigger concern: their actual death toll in Germany does not match the raw data from the German statistical offices, which are also collected in projects such as the ADM. This mismatch affected not only the deaths reported in 2020 and 2021, but also the historical data for 2015-19. This had played an important role in its low extrapolation of expected deaths.
The mismatch occurred because WHO scientists had adjusted – or ‘scaled’ – the gross mortality data. The WHO often does this with data it receives from countries, Wakefield says. This may be for a good reason: the organization is trying to adjust to insufficient reports, inconsistencies with other data flows, or “completeness” errors, when mortality data for the last few months is expected to increase as get more results, for example. But it was less clear that this process would apply to Germany, a country with detailed mortality reports. “We need to look at how the adjustment for the sub-report is done,” Wakefield says.
The Wakefield team returned to raw data and used a linear extrapolation to 2020 and 2021 (see “Revised Death of Germany”). Ironically, spline extrapolation to raw data yields similar results, Wakefield adds. The overall effect reduces estimates of Germany’s excess deaths in 2020-2021 from 195,000 to 122,000 (ranging from 101,000 to 143,000). Its excessive mortality rate drops to 72.7 per 100,000 people per year, compared to 116 per year in the previous WHO report.
Fonts: World Mortality Dataset / WHO / J. Wakefield
Researchers also corrected WHO numbers for Sweden, following similar criticism. One group that took part was the COVID-19 Actuaries Response Group, a forum of mostly UK-based actuaries who have regularly examined the impact of the pandemic on mortality. On May 16, the group released a blog stating that Sweden’s death toll in the WHO report does not match Statistics Sweden’s (go.nature.com/3nctx5s). In fact, WHO figures seemed to differ from official sources of information for many European countries.
They turned out to be the same two problems: the spline technique and the scale. Again, the Wakefield team revised its approach to using linear extrapolation to raw data (see “Revised Death of Sweden”). In this case, Sweden’s annual excess mortality went from 55.8 to 66.1 per 100,000. Schöley blames Wakefield for tackling the problem quickly. “A model to follow on how to deal with an honest peer review,” Schöley tweeted.
Fonts: World Mortality Dataset / WHO / J. Wakefield
Other European countries could also be affected by the escalation of WHO death toll: Norway is another that critics have questioned. Wakefield says his team will now review WHO scale procedures, as well as how it is extrapolated from historical data. “I don’t think it makes a big difference for most countries,” he says.
The revisions bring the WHO figures for Germany much closer to those of another model, produced by The economist. But a third model, from the Institute for Health Metrics and Evaluation (IHME) in Seattle, Washington, published in The Lancet, is now an atypical value2. It is estimated that more than 200,000 deaths are in excess for Germany. “The economist The method is the most transparent and defensible, “the Wakefield team wrote in their technical paper.
When asked for a comment, Haidong Wang, a demographer and population health specialist at the IHME, responded only that differences in overall estimates may arise from how models treat mortality data, including how they adjust the figures for under-registration and completeness issues, and how the models estimate expected mortality.
How to compare countries
As soon as the results came out of the WHO, researchers, politicians, …