Why Small Effects Fail: An Interactive Tool for Understanding Predictive Utility
Poster No:
1160
Submission Type:
Late-Breaking Abstract Submission
Authors:
Povilas Karvelis1, Andreea Diaconescu2
Institutions:
1CAMH, Toronto, MT, 2Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health, Toronto, ON
First Author:
Co-Author:
Andreea Diaconescu
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health
Toronto, ON
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health
Toronto, ON
Late Breaking Reviewer(s):
Eduardo Garza-Villarreal, M.D., Ph.D.
Universidad Nacional Autónoma de México campus Juriquilla
Juriquilla, Querétaro
Universidad Nacional Autónoma de México campus Juriquilla
Juriquilla, Querétaro
Introduction:
Clinical predictions require biomarkers with large effect sizes, yet effect sizes are rarely interpreted based on their predictive power. This leads to overinterpretation of small effects, unrealistic expectations about their potential impact and inefficient resource allocation. To address this, we introduce an interactive web-based tool that helps researchers develop intuition for how statistical effect sizes translate into predictive performance metrics relevant for clinical decision-making.
Methods:
Our web-based tool provides visualizations and converts commonly reported effect sizes (e.g., correlation strength, standardized mean differences) into predictive metrics such as area under the curve (AUC), balanced accuracy, and positive predictive value. By allowing users to adjust the reliability of measures and the base rates of outcomes, the tool intuitively demonstrates how effect size constraints shape predictive utility in clinical settings.
To complement this tool, we also implement a calculator for the Mahalanobis distance D, which is a multivariate generalization of Cohen's d. This provides a way to explore how multiple predictors of smaller effect sizes can be combined to achieve a larger group separation in multivariate space. Mahalanobis D can provide a good approximation of the performance of machine learning models when the predictors are normally distributed and do not have strong interactions.
To complement this tool, we also implement a calculator for the Mahalanobis distance D, which is a multivariate generalization of Cohen's d. This provides a way to explore how multiple predictors of smaller effect sizes can be combined to achieve a larger group separation in multivariate space. Mahalanobis D can provide a good approximation of the performance of machine learning models when the predictors are normally distributed and do not have strong interactions.
Results:
We illustrate the tool's utility in three scenarios: screening for mental health risks (psychosis onset, suicide attempt), diagnosing mental disorders (depression), and predicting treatment response (to antidepressants). The results show that clinically useful predictions require effect sizes far larger than those typically observed, particularly in conditions with low base rates (Fig 1). Additionally, we demonstrate how poor measurement reliability constrains effect sizes, underscoring the need for improving assessment reliability in psychiatric research.
Using the Mahalanobis D calculator, we further show that it is much easier to achieve larger group separation in multivariate space with a handful of powerful predictors rather than with dozens of small predictors with small effect sizes (Fig 2). With many predictors, the presence of even small amounts of collinearity quickly adds up to limit the resulting Mahalanobis D.
Using the Mahalanobis D calculator, we further show that it is much easier to achieve larger group separation in multivariate space with a handful of powerful predictors rather than with dozens of small predictors with small effect sizes (Fig 2). With many predictors, the presence of even small amounts of collinearity quickly adds up to limit the resulting Mahalanobis D.
Conclusions:
Overall, our tool provides a first-principles approach to interpreting effect sizes in real-world contexts. By making the limitations of small effect sizes explicit, this tool serves as a practical resource for study planning and hypothesis formulation, while highlighting the need to improve effect sizes and reliability of measures.
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Methods Development 2
Other Methods
Keywords:
Modeling
Open-Source Software
Statistical Methods
1|2Indicates the priority used for review