MINI CASE 14.1

The use of significance testing is an important part of the research process. It provides a rigorous way to judge our findings and the work of others. There is a need to specify an hypothesis, make a decision on the significance level and apply an appropriate test. The process is transparent and can be examined for validity. All well and good in theory!

In the last few chapters we have only offered you a few of the statistical tests available. You only need to look at a source like en.wikpedia.org and search on 'non parametric tests' to see how many are available. It can be difficult to know which test to use for some of the problems encountered in practice.

It is also the case that if you keep looking then eventually you will find something of statistical significance. Given the typical choice of a 5 per cent significance level, you should find significance 1 in 20 times even if it does not exist. The following is a reminder, that the results you present are only as good as your research methods and integrity

The cult of statistical significance

Krämer in his paper 'The cult of statistical significance' suggests that a claim of statistical significance can be compromised by: (a) the size and generation of the sample and doing lots of tests and reporting only the most 'significant' results; (b) the empirical improprieties of HARKing (Hypothesizing After the Results are Known); and (c) mistaking a rejected null hypothesis as proof that the alternative is true. Reporting of results can also be further compromised by publication bias whereby research which yields non-significant results is not published. This can result in other independent researchers repeating an investigation until eventually by chance a significant result is obtained. Fanelli in her paper 'Do Pressures to Publish Increase Scientists' Bias?' suggests that given a 'negative' result a researcher 'might be tempted to either not spend time publishing it (what is often called the "file-drawer effect", because negative papers are imagined to lie in scientists' drawers) or to turn it somehow into a positive result'. This can be done by re-formulating the hypothesis (Harking), by selecting the results to be published, or by tweaking data or analyzes to 'improve' the outcome.

To highlight point (c) above, Krämer cites a meta-analysis that combined various previous research into childhood leukemia. The analysis purports to show that nuclear power plants induce a high risk of leukemia by rejecting a null hypothesis as proof that the alternative is true. The alleged significance appears to be mostly due to reporting only significant results, publication bias and a disregard of important confounding factors: race (risk of childhood cancer is almost double for whites compared with blacks); gender (risk of childhood cancer, in the USA, is 30 per cent higher in boys); income (incidence of leukemia is almost double in children from richer families). One of the sets of data was derived in an area in close proximity to a nuclear power plant as well as having almost all of the confounding factors that correlate strongly with childhood leukemia. This one area contributed to almost all of the cases 'on which the ''significant'' increase of childhood leukemia in the vicinity of nuclear power plants is based'. Removal of this areas data set and adding others that were initially overlooked resulted in the retention of the null hypothesis. This example highlights that one needs to be aware of spurious significance due to bad practice, also that claims of significance of any sort require that the underlying model be reasonably correct.