Go Figure: Keep context in images to meaningfully interpret results
Poster No:
1566
Submission Type:
Abstract Submission
Authors:
Paul Taylor1, Peter Bandettini1, Cesar Caballero-Gaudes2, Vince Calhoun3, Jen Evans4, Daniel Glen5, Javier Gonzalez-Castillo1, Omer Faruk Gulban6, Daniel Handwerker1, Peter Lauren1, David Leopold7, Amanda Mejia8, Cyril Pernet9, Luiz Pessoa10, Justin Rajendra1, Richard Reynolds11, Vinai Roopchansingh1, Brian Russ12, Salvatore Torrisi13, Gang Chen7
Institutions:
1National Institute of Mental Health, Bethesda, MD, 2Basque Center on Cognition, Brain and Language, San Sebastian, Spain, 3GSU/GATech/Emory, Atlanta, GA, 4NIH, Bethesda, MD, 5National Institute of Mental Health (NIMH), NIH, Bethesda, MD, 6Maastricht University, Maastricht, Netherlands, 7National Institutes of Health, Bethesda, MD, 8Indiana University, Bllomington, IN, 9Copenhagen University Hospital, Copenhagen, Denmark, 10University of Maryland, College Park, MD, 11NIMH, Bethesda, MD, 12Icahn School of Medicine at Mount Sinai, New York, NY, 13University of California, San Francisco, San Francisco, CA
First Author:
Co-Author(s):
Introduction:
Conventional neuroimaging figures present a small number of regions after strict thresholding and base all remaining interpretations on those few arbitrarily filtered clusters. In this "opaque thresholding", subthreshold results from the rest of the brain are totally hidden and withheld from consideration. This focus on sparse localization originated from early neuroimaging studies, but modern paradigms often investigate networks and cross-region relationships, with a view that most functions involve the interaction of many parts of the brain and to varying degrees. While regions with high statistical strength might still be most important, they are not acting alone: other parts of their, or separate, networks can be involved with weaker statistical evidence. Subthreshold results provide key context to understand significant clusters and wider brain functionality; they should be included, e.g. with transparent thresholding [1,2]. We demonstrate these points and the benefit of retaining context in images with real-world examples.
Methods:
We present examples of the importance of keeping context in brain images. Ex 1 uses group results from Flanker task FMRI data, see [3] for processing in AFNI [4]. Ex 2 uses group results from Hyp 2&4 of the public NARPS data [5], see [2] for processing in AFNI. For each, opaque and transparent thresholding was applied with voxelwise p=0.001 and cluster familywise error FWE=5%.
Results:
Ex 1: Fig 1A shows opaque thresholded results, which contain the lobes of a single cluster in the right intraparietal sulcus in the slice. Even a cluster with just one voxel below cut-off would be hidden. We can only make biological interpretations from this sparse view: the effect appears to be strongly lateralized in this region, with no other effects present. Fig 1B shows transparently thresholded results, revealing nontrivial effects in brainwide GM, many of which seem biologically relevant. The "lone cluster" actually appears to be much more left-right symmetric.
Over-reducing results has been studied in statistics. Fig. 1C shows Anscombe's Quartet [5], a classic example: the 4 sets of points have the same summary statistics and correlation value, but very different underlying data patterns. Only by adding in the contextual information of the plot can the ambiguity of the summary values be resolved. Identical reasoning applies to neuroimaging. Fig. 1D shows 4 images of contextualized results. Each reduces to Fig 1A with opaque thresholding but their contexts have very different biological implications: 1) fairly strong left-right symmetry (actual effects); 2) left-right anti-symmetry; 3) strong right-lateralization; 4) likely noise or artifact. Transparent thresholding retains necessary context.
Ex. 2: Fig. 2 displays two-sided t-test results with opaque thresholding, where the number of subjects changes slightly in each row. Both the coverage and number of clusters vary notably, and changes are not just convergent: one cluster disappears and then reappears. A similarity matrix of clusters shows the large variability across an extended set of group sizes (down to 17). Opaque thresholded interpretations are very sensitive to group size.
Fig 2B displays the same results using transparent thresholding. The images are much more consistent, reflecting less sensitivity to arbitrary group size differences. Cluster count changes are still known, but the contextualized maps allow for the reader's evaluation to remain more consistent and stable. The similarity matrix here remains uniformly quite high even down to the smallest group size. Additional benefits of transparency include knowledge of the context itself.
Over-reducing results has been studied in statistics. Fig. 1C shows Anscombe's Quartet [5], a classic example: the 4 sets of points have the same summary statistics and correlation value, but very different underlying data patterns. Only by adding in the contextual information of the plot can the ambiguity of the summary values be resolved. Identical reasoning applies to neuroimaging. Fig. 1D shows 4 images of contextualized results. Each reduces to Fig 1A with opaque thresholding but their contexts have very different biological implications: 1) fairly strong left-right symmetry (actual effects); 2) left-right anti-symmetry; 3) strong right-lateralization; 4) likely noise or artifact. Transparent thresholding retains necessary context.
Ex. 2: Fig. 2 displays two-sided t-test results with opaque thresholding, where the number of subjects changes slightly in each row. Both the coverage and number of clusters vary notably, and changes are not just convergent: one cluster disappears and then reappears. A similarity matrix of clusters shows the large variability across an extended set of group sizes (down to 17). Opaque thresholded interpretations are very sensitive to group size.
Fig 2B displays the same results using transparent thresholding. The images are much more consistent, reflecting less sensitivity to arbitrary group size differences. Cluster count changes are still known, but the contextualized maps allow for the reader's evaluation to remain more consistent and stable. The similarity matrix here remains uniformly quite high even down to the smallest group size. Additional benefits of transparency include knowledge of the context itself.
Conclusions:
Informative figures are highly critical for modern neuroimaging. Transparent thresholding retains useful context for interpretation and reduces hypersensitivity to arbitrary changes. The neuroimaging field would greatly benefit from including both thresholded and subthreshold information in reported results.
Modeling and Analysis Methods:
Methods Development 1
Other Methods 2
Keywords:
Data analysis
Design and Analysis
FUNCTIONAL MRI
Workflows
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Are you Internal Review Board (IRB) certified? Please note: Failure to have IRB, if applicable will lead to automatic rejection of abstract.
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
[2] Taylor PA, Reynolds RC, Calhoun V, Gonzalez-Castillo J, Handwerker DA, Bandettini PA, Mejia AF, Chen G (2023). Highlight Results, Don’t Hide Them: Enhance interpretation, reduce biases and improve reproducibility. Neuroimage 274:120138.
[3] Chen G, Pine DS, Brotman MA, Smith AR, Cox RW, Taylor PA, Haller SP (2022). Hyperbolic trade-off: the importance of balancing trial and subject sample sizes in neuroimaging. NeuroImage 247:118786.
[4] Cox RW (1996). AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res 29(3):162-173. doi:10.1006/cbmr.1996.0014
[5] Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, et al. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582(7810):84-88.