Certainty in complex scientific research an unachievable goal

03-Feb-2017 - Canada

A University of Toronto study on uncertainty in scientific research could shed light on anomalies that arose in early attempts to discover the Higgs boson and even how polls failed to predict the outcome of Donald Trump winning the U.S. presidential election.

The study suggests that research in some of the more complex scientific disciplines, such as medicine or particle physics, often doesn't eliminate uncertainties to the extent we might expect.

"This is due to a tendency to under-estimate the chance of significant abnormalities in results." said study author David Bailey, a professor in U of T's Department of Physics.

Looking at 41,000 measurements of 3,200 quantities - from the mass of an electron to the carbon dating of a sample - Bailey found that anomalous observations happened up to 100,000 times more often than expected.

"The chance of large differences does not fall off exponentially as you'd expect in a normal bell curve," said Bailey.

A long tail of uncertainty

"The study shows that researchers in many fields do a good job of estimating the size of typical errors in their measurements, but usually underestimate the chance of large errors," said Bailey, noting that the larger-than-expected frequency of large differences may be an almost inevitable consequence of the complex nature of scientific research.

"As measurements become more and more accurate, the smallest things matter more and more," Bailey said.

"If two measurements agree, you're happy. If not, you see there's something you need to investigate," he said. "You track down the cause of the variation and report the cause or you say that you don't know the cause and this reduces the trust in your result."

But with finite time and financial resources, researchers often have to make a choice between having a large sample of data, such as tens of thousands of people in a survey, and having a large number of variables you want to understand.

"You start with a very large sample that just lumps everyone together. You then might have to ask if your result is the same for both men and women. Is it the same for different backgrounds, Canadians versus Americans, for example," says Bailey. "At that point, you have to ask if your results hold for the smaller data set. Your sample is getting smaller and more can go wrong."

Impossible not to be a little wrong?

Physics studies did not fare significantly better than the medical and other research observed. However, the highly quantifiable way in which values and uncertainties are reported, may make physics more useful in terms of the degree of reproduce-ability of results that researchers should reasonably expect.

"Scientists will still aim for the most accurate results, but their expectations of how well those aims are met may be tempered in light of this research," said Bailey.

He believes his study can help researchers better analyze their data, motivate more care with novel results, and encourage more realistic expectations by both scientists and the public about the accuracy of scientific research.

"These insights can be beneficial given the inherently complex nature of scientific research," says Bailey. "But the chance of avoiding being wrong in some way on some level is almost impossible."