Check out this answer from Consensus:
While p-values have their limitations and are often misused, they still hold value in certain research contexts, particularly where dichotomic answers are needed. However, researchers should consider augmenting or replacing p-values with alternative statistical methods like confidence intervals, effect sizes, Bayes factors, and Akaike Information Criteria to gain a more nuanced understanding of their data. Improved education and training in data analysis are also crucial to address the deeper issues in statistical practices.
The p-value has been a cornerstone of statistical analysis, particularly in fields like biology and clinical research. However, its utility and interpretation have come under scrutiny, leading to debates about whether researchers should continue to focus on p-values or adopt alternative methods.
Key Insights
Misuse and Limitations of P-values:
P-values are often misinterpreted and can be manipulated by changing data analysis methods, leading to arbitrary levels of statistical significance1.
The American Statistical Association has highlighted the misuse of p-values and suggested that simply reducing the significance threshold (e.g., from 0.05 to 0.005) may not solve the underlying issues2.
Need for Dichotomic Answers in Clinical Research:
In clinical research, particularly in diagnostic imaging and interventional radiology, dichotomic answers are crucial for decision-making. Therefore, a conservative approach to p-values is recommended2.
Alternative Statistical Approaches:
Confidence Intervals and Effect Sizes: Augmenting p-values with information about confidence intervals and effect sizes can provide a more comprehensive understanding of the data3.
Bayes Factor: Using Bayes factors can offer a relative measure of evidence for null and alternative hypotheses, which is useful for studies that collect data until clear evidence is obtained3.
Akaike Information Criteria (AIC): For model-building studies, AIC can replace p-values by providing quantified information on the best model3.
Education and Training:
Better education and training in data analysis are essential. Researchers need to be trained in relevant software and concepts to understand the outputs and potential problems with an analysis1.
Should researchers stop focusing on the p-value?
Sin-Ho Jung has answered Unlikely An expert from Duke University in Biostatistics, Bioinformatics
I don't know why use of p-value can be an issue. P-value is just a measure of significance. Although it can't replace all other analysis methods it can't be replaced by other measures either. The only possible issue associated with p-value is to dichotomize it at 0.05, considering one larger than 0.05 as insignificant. It is a continuous measure and 0.05 can't be the only cutoff.
Should researchers stop focusing on the p-value?
Lewis Halsey has answered Near Certain An expert from University of Roehampton in Behavioural Science, Physiology
My own work highlights a major problem with the p-value - it's 'fickleness'. That is, the p-value of an experiment will tend to vary greatly from one replicate of the experiment to another. This means that while on one occasion p can be very high, on another occasion p can be very low. P is unstable unless the power of the study is incredibly high (around 90%+) which is very rare. Colleagues and I have written about this: https://www.nature.com/articles/nmeth.3288
Should researchers stop focusing on the p-value?
Brett Lidbury has answered Unlikely An expert from Australian National University in Pathology, Statistics
The p-value can remain as part of the statistical toolkit, but must be used as a component within broader statistical analyses, and not as the "be-all-and-end-all". A good statistical analysis must also include an effect size measure, and calculation of statistical power where appropriate. Using the correct tests appropriate for the data distribution must also be done (e.g. K-S statistic). For multiple hypotheses testing, a Bonferroni correction, or similar, is required. If these efforts are made and interpreted correctly, there's no reason why the p-value cannot continue to be useful. I agree though, accepting the p-value uncritically is a big problem today. The way ahead - predictive models (accuracy estimates and kappa statistics).
Should researchers stop focusing on the p-value?
David Eccles has answered Unlikely An expert from Malaghan Institute of Medical Research in Genetics, Genomics, Bioinformatics, Biostatistics, Data Science
It's important for researchers to be aware of what p-values mean. They are useful in combination with other measures, but should never be used on their own.
The American Statistical Association has produced an excellent position paper on the p-value, which attempts to explain p-values by six principles:
P-values can indicate how incompatible the data are with a specified statistical model.
P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
Proper inference requires full reporting and transparency
A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
Point 5 seems to be the thing that researchers have the most trouble getting their head around. Any research that ranks results by their p-value is using the p-value as a measure of the importance of a result. Almost all genome-wide association studies that I've come across have this issue, showing genome-wide manhattan plots with the y-axis of the graph as the p-value (with the interpretation that p-value peaks are important), and ranking associated variants by their p-value, rather than the effect size (or some other measure).
My general recommendation is to take the p-value into account, but use it as an initial filter only: discard results that fit the expected null-distribution model (or don't fit, depending on the approach), then remove the p-value column from the statistics for subsequent analysis, so that it is much less likely to be misused.
Should researchers stop focusing on the p-value?
Robert Newcombe has answered Near Certain An expert from Cardiff University in Biostatistics The p-value isn't entirely useless, but shouldn't be regarded as the primary communication of research findings. It has to do with the plausibility of chance or coincidence as the explanation for an observed difference between two means or such like. Thus it is highly sample-size-dependent. It is far more informative to give a numerical estimate of the difference, in the relevant units, accompanied by a confidence interval to show the impact of sampling uncertainty. Sometimes we obviously want to report either an absolute difference (between two means, say, or between two proportions). Sometimes, a ratio (such as a relative risk) or a closely related value such as a relative risk reduction. There are other useful effect size measures, such as U/mn, the Mann-Whitney U statistic divided by the product of the two sample sizes, that ranges from 0.5 (when two series are identical) to 0 or 1 (when they don't overlap). Any of these can be appropriate to report, with a confidence interval. Each of these can be interpreted in its own right, without specific reference to the sample size. This forms a much better basis for communicating research findings than a paradigm in which 'p<0.05' is taken to mean 'real' and 'p>0.05' is taken to mean 'null'. For more explanation, see my book, Confidence Intervals for Proportions and Related Measures of Effect Size available at http://www.crcpress.com/product/isbn/9781439812785
