We’ve all done it — felt a cough, headache or fever coming on then searched online for a remedy. If you’re like most people, you’re probably confident you can assess the effectiveness of treatments you find on the web, separating medically beneficial ones from those that are a waste of money, dubious or even harmful.
But a recent study conducted by PhD candidate Amira Ghenai and her colleagues in the Data Systems group at the Cheriton School of Computer Science should give pause to anyone who thinks assessing online health information is straightforward.
To answer this, Ghenai and her colleagues conducted a controlled study on 60 University of Waterloo students in which the search results participants saw were intentionally biased toward correct or incorrect information for 10 medical treatments.
“We chose treatments that are commonly searched for, such as is cinnamon helpful for diabetes, are insoles helpful for back pain, does caffeine help alleviate asthma,” she said, adding that they asked participants to pretend they had a question about the effectiveness of a medical treatment and they were going to use a search engine to help them answer it.
For each treatment, participants were presented with a search results page containing 10 results modelled after what people would see if they had used Google or Bing — i.e., each search result had a document title, a snippet of text and a URL to an actual page.
“To bias search results towards correct information, we selected eight of the results to be correct and two to be incorrect,” Ghenai explained, adding that a correct result was a document that contained information about the efficacy of a medical treatment that supports the truth, and an incorrect result contained information that contradicted established medical knowledge.
To bias search results towards incorrect information, they did the opposite — selected eight results to be incorrect and two to be correct. As a control, participants were asked to answer whether a treatment was helpful, inconclusive or unhelpful without being shown any search results.
“We found that search results have a significant effect on people’s ability to make correct decisions. When search results were biased towards correct information it increased participant accuracy from 43% to 65%. And when search results were biased towards incorrect information it cut participant accuracy almost in half, from 43% to 23%. In short, bad information is worse than no information at all.”
The influence of incorrect information is concerning because sometimes the condition for which people are seeking treatment is much more serious than the efficacy of insoles for back pain and the outcome of an incorrect decision can be grave.
“When people use search engines to find answers to health questions, their interaction with the results has the potential for both positive and negative outcomes,” she explained. “This can be dangerous because search engines return results that are relevant to your query, not necessarily results that have correct information.”
Perhaps the most infamous example of misleading search results is that experienced by Wei Zexi, a 21-year-old Chinese student, who died from synovial sarcoma, a kind of cancer. In the early stages of his illness, doctors used chemotherapy and radiation — conventional cancer treatments — but when they were no longer helping, Zexi looked online for other options. He found a hospital that claimed to offer an experimental treatment for his cancer, but after travelling to the facility and having the expensive treatment, which cost his family 200,000 yuan — about $41,000 CDN at today’s exchange rates — he discovered the treatment was neither effective nor experimental.
This story is notable because Zexi found the hospital offering the treatment using Baidu, a search engine commonly used in China. He later learned from friend using Google outside China that there was no scientific evidence the treatment would help. Zexi had found the ineffective treatment on a Baidu search results page, but it was not revealed was that the search result was a paid advertisement rather than the result of an organic search.
“When people find medical treatments or information that can improve their life or that of a loved one, search engines can make people’s lives better. But when search engines mix correct with incorrect information, we have shown there is potential for significant harm. And when people are desperate they might see what they want to believe.”