Bettering scientific reporting by replacing p values with simple, informative, objective and flexible Bayes factors

Speaker

Klaus Rostgaard, Senior Statistician, Danish Cancer Society

Abstract

This talk is about the situation where we (in principle) have a d‐dimensional parameter estimate of interest and the d x d dimensional (co‐)variance of it and make inference from that. Often, we test a null hypothesis H0: that the parameter of interest is 0, versus the alternative H1: that the parameter can be anywhere in the d-dimensional parameter space. The testing mindset has many unfortunate behavioural side effects on what is reported and how in the scientific literature. Furthermore, traditional significance testing, i.e., using p values, does not suffice to compare the evidence in favour of H0 and H1, respectively, as it only makes assumptions about H0. In practise, it is therefore biased in the direction of favouring H1. Bayesian methodology takes H1 into consideration, but often in a way that is either subjective (contradicting scientific ideals) or objective by way of assuming very little information in the prior, which by itself is untrustworthy and often clearly favours H0.

Here we develop an approximation of the so‐called Bayes factor (BF) applicable in the above setting; BF is the Bayesian equivalent of a likelihood ratio. By design the approximation is monotone in the p value. It it thus a tool to transform p values into evidence (probabilities of H0 and H1, respectively). This BF depends on a parameter that expresses the ratio of information contained in the likelihood and the prior. We give suggestions for choosing this parameter. The standard version of our BF corresponds to a continuous version of the Akaike information criterion for model (hypothesis) selection.<br><br>Posterior odds of H1 and H0, i.e., Pr(H1|X)/Pr(H0|X) (and hence probabilities and evidence for each), are obtained by multiplying BF with prior (pre-data) odds of H1 and H0, i.e., Pr(H1)/Pr(H0). We suggest that for scientific reporting and discussion prior odds should be set to 1; the reader can modify prior odds to fit their own a priori beliefs and obtain the corresponding posterior inferences. BF=1 represents equiprobability of the hypothesis, H0 and H1. BF is thus centered at the right value, for the purpose of making immediate judgments about which hypothesis is the more likely and how strong the evidence is for that based only on the likelihood function.

Replacing p-values (and implicit tests by confidence intervals) by BFs should allow for shorter, more informative, and less biased reporting of many scientific studies. We exemplify the calculations and interpretations and illustrate the flexibility of our approach based on a real-world epidemiologic example where we a priori believe H0 to be a good approximation of physical reality. H0 is that an 8-dimensional predictor has exactly the same non-trivial effect (measured by a hazard ratio) on two distinct disease outcomes. Finally, we compare these new BF-based inferences with those based on p values. Although there is a bijection between BF and p for fixed d it is non-trivial – so you need to calculate BF. Generally, BF-based inference is more in favour of H0 than p-value inference, i.e., less biased in favour of the alternative, H1. The BF is easy to calculate (only requires d and p or a test statistic), flexible and objective. It is a Bayesian solution to the Fisherian project of making statistical inference based exclusively on the likelihood function.

More information

For more information about upcoming and past seminars go to https://biostatistics.dk/seminars/

Unless otherwise stated, seminars will be held at CSS (det gamle Kommunehospital), Øster Farimagsgade 5, 1353 Copenhagen K, room 5.2.46. Tea will be served in the library of the section of Biostatistics half an hour before the seminar starts.