Yahoo Movies Victorian government used ‘low grade’ mask study to justify mandate, experts say
A study relied on by the Victorian government to justify its strict mask mandate has been criticised by some doctors and epidemiologists as “low grade” evidence.
While the experts emphasised that wearing masks indoors was important and that masks helped reduce transmission, they said the “world first” joint study by the Burnet Institute and the Kirby Institute was flawed.
The study, published in the medical journal Plos One in July, used newspaper photographs and surveys to assess mask compliance and its effect on Covid rates.
The mask mandate in Victoria has been controversial because it requires people to wear masks at all times, including when outdoors in open spaces, despite strong evidence that the risk of indoor transmission, especially within households, is significantly higher.
In August the premier, Daniel Andrews, announced Victorians would also be barred from removing their mask to drink alcohol outdoors, though it can be removed to eat or to drink non-alcoholic beverages. The Plos One study has been used by the government to justify its policy, which applies even when people are walking outside with no one nearby.
The study analysed images of people in public places from the digital archive of The Age newspaper taken between 10 July and 2 August 2020 – before and after the mandatory mask policy was introduced in the state’s long lockdown.
From these, all photos taken in public locations in urban Melbourne, such as streetscapes and shopping centres, were used to calculate the proportion of people wearing masks in public across three time periods: 10–19 July (the period preceding mandatory mask policy announcement); 20–22 July (period between announcing and commencing mandatory mask wearing); and 23 July to 2 August (mandatory mask wearing period). The researchers also conducted a survey.
The paper found that “mandatory mask use policy substantially increased public use of masks and was associated with a significant decline in new COVID-19 cases”.
A Burnet institute press release said the “world-first” study had found mask-wearing turned the pandemic around “almost overnight” and were the single most important tool in turning the tide of Victoria’s second Covid wave in 2020.
“Importantly given the growing evidence of outdoor transmission of the Delta variant, it makes sense that masks are worn both indoors and outdoors during a Covid-19 outbreak,” an author of the paper said in a press release promoting the study.
But the study has been called into question after news.com.au published concerns about the paper’s methodology. The news.com.au report said the Victorian health department had pointed to the study as evidence to support the mandate.
Researcher Dr Kyle Sheldrick, who first raised concerns about the paper, said there were numerous methodological problems with the study.
“Drawing photographs from those taken by news photographers from a particular publication as a basis for causal inference is staggering,” Sheldrick said. “News photographers don’t go out and just snap a random sample of people in the community, they are tasked with photographs for particular stories and editorial lines. It’s hard to think of a more biased sample.”
Sheldrick said there was a mismatch between the methods described and the results reported, and has detailed his concerns publicly.
“Despite indicating that the archivist at the newspaper checked these two 14-day periods for additional unpublished photographs, which adds very little to be honest, and analysed all the resulting images, the results of that search are never described, and the authors in their results describe only 44 published images,” he said.
The infectious diseases physician Prof Peter Collignon agreed with Sheldrick and said observational studies were among the weakest types of study. This is because they often cannot prove cause and effect; and there are too many “confounding factors” that are difficult to control for, but which could influence the results.
“I think it was a mistake to use this study for any policy,” Collignon said.
“I think there are so many potential flaws in the methodology that this is not high-grade evidence. If you want high-grade evidence, you need to do randomised, controlled, double-blind studies. Well, this study isn’t it, and by any evidence criteria would be considered low-grade.”
An epidemiologist at the University of Wollongong, Gideon Meyerowitz-Katz, said the study “is not sufficiently robust to determine a causal connection between the mask mandate, masking and a decline in Covid cases”.
“While the authors used two methods to derive an estimate of the change in masking behaviour, neither of these gave a clear indication of how many people were wearing masks in the city or state over time,” he said. “The photos are media shots and not representative of the population, and the survey was only conducted over a short time period and it included only 18 people after the 23rd of July.”
However, he said a randomised control trial from Bangladesh had found mask policies can have a reasonable impact on cases, in the order of a 10% reduction. “So it’s important to note that flaws in one study don’t mean that masks don’t work,” he said.
In a statement the Burnet Institute, which provides modelling to the Victorian government, described Sheldrick as “a PhD student” and said it stood by the study.
“We accept that not everyone has to agree with our work … Others, including the peer-reviewers of our paper thought differently,” the statement said.
“Our paper … showed that the introduction of the mandatory mask policy in Melbourne coincided with a marked decline in cases. The introduction of the policy was introduced in isolation from other restrictions which allowed for a unique assessment of its potential. This was unusual and why we claimed it was a ‘world first’ at the time.”
Sheldrick said he was disappointed by the institute’s response.
“The response doesn’t address specific criticisms like the fact all the ‘before’ photos are taken in the later afternoon, all the ‘after mandate’ photos are taken before lunch, so there’s no way to actually compare the two meaningfully,” he said.
The chair of epidemiology at Deakin University, Prof Catherine Bennett, said the study was a useful contribution. While she acknowledged the researchers had examined photographs and data from “a very small window” of time and had assumed a linear relationship between masks and reduced Covid-cases, “which is a bit unusual for an epidemic curves”, they had shown the mask policy had translated to the public adopting the measure.
“Tracking it and making sure that the policy does translate to a practice change in the community is a useful contribution,” she said.
