Important stories hidden in Google's 'experiment' blocking Australian news sites

Google’s “experiment” in Australia to remove major news sites from search results is hiding important news stories from hundreds of thousands of Australians.

In some cases filtering out mainstream news publications from search results is also resulting in lower-quality publications being promoted, including a news website known for spreading misinformation and conspiracy theories.

The search giant previously admitted that it is running “a few experiments that will each reach about 1% of Google Search users in Australia to measure the impacts of news businesses and Google Search on each other”, but has previously refused to answer detailed questions about the experiment.

Related: What are my options if Google shuts down search in Australia?

For some users, major news sites will not show up in search results, and users are not informed that their search results are being restricted. People are still able to view the news stories in Google news, and can find news stories by normal methods such as visiting news websites directly or using alternative search engines.

The experiment does not remove results that link to official information, such as that found on government websites.

A Google spokesperson confirmed that a “small per cent of users” would not see some news stories in Google search, but disputed that this would prevent Australians from reaching news sources.

“It is demonstrably incorrect to claim this has prevented Australians from reaching news sources. First, these are tiny experiments, impacting only a very small per cent of users, so the vast majority of users are totally unaffected,” he said.

“Second, even users affected by the experiments can continue to access the news websites, for example they can go to publishers’ websites directly, use publishers’ apps, via Google News, follow links on social media or shared by friends and family, or get news from TV, radio, print newspapers and magazines.”

Guardian Australia was able to generate browser sessions that were affected by news site removal, and can reveal the impact of blocking news sites compared with a normal search session.

A comparison of two side-by-side search sessions shows that in this case the “experiment” removes results for major Nine Entertainment websites such as the Sydney Morning Herald, the Age and the AFR; News Corp Australia websites, including major news sites like the Herald Sun, but also including regional publications such as the Gympie Times; the Guardian’s website; and Seven West Media websites such as 7news and the West Australian.

Even searches like “the Age” or “the Herald Sun” will not return the main news websites in search results.

Other news sites such as the ABC, BBC and Canberra Times were unaffected. The ABC was reportedly filtered out of search results for a user last week, so it’s possible the difference in news sites is down to different “experiments”.

This filter results in users being unable to see some news stories entirely in search results if it is not being covered widely. For example, Guardian Australia’s exclusive story about allegations of mistreatment of an Indigenous woman in an ACT prison did not show up at all in search results affected by the filter.

The Sydney Morning Herald’s story about increasing donations from property developers was similarly affected, and the original story did not show up at all in searches affected by Google’s restrictions, only mentions of the story on social media and other websites.

For other news stories, such as this story about Asic, the corporate regulator being targeted by hackers, the filter results in the top news results being from sources outside of Australia.

In some cases, the experiment has resulted in sources of questionable quality being promoted over the blocked news websites. A search for “Scott Morrison” conducted last Thursday by a colleague who was experiencing the restriction on news sites returned the Epoch Times as one of the top three results. The Epoch Times has reportedly spread pro-Trump conspiracy theories, QAnon content, and anti-vaccine misinformation.

Google has said that the experiment will reach around 1% of users. However, given the large market share of Google in Australia (about 95%), and the high proportion of Australians who use the internet for news purposes (at least 86% for those over 18), this experiment could feasibly affect at least 160,000 people based on a back-of-the-envelope calculation.

default

A spokesperson from Google said the experiment related to its ongoing involvement in the development of the Australian government’s draft bargaining code that would force them to negotiate a fair price for displaying local news content.

“We remain committed to reaching a solution for a workable code, as we have been throughout this whole process. And the small, short-term experiments we are currently running are part of our preparations for the code, which among other things, may require us to demonstrate the impact of news on our platform,” he said.

Google has previously criticised the proposed code in messages to the search engine’s users while Facebook has warned it could block Australians from sharing local news as a consequence.

• Note: Guardian Australia has been in discussion with Google over inclusion in its new “Showcase” offering that would see the company pay news providers for inclusion in this product.