Decoding representations of common objects using fMRI

Abstract

Decoding object-specific representations in functional magnetic resonance imaging (fMRI) can support numerous applications, including clinical treatments of specific phobias and post-traumatic stress disorder (PTSD). For example, through decoded neurofeedback (DecNef), pairing representations of commonly feared animals with monetary rewards can reduce physiological fear responses. To maximize the effectiveness of treatment, one must have a good decoder to classify fMRI signals. This thesis presents the methods to improve the fMRI decoding performance of common objects by improving the between-subject alignment, feature selection, and classification methods. The main dataset used here consists of fMRI images of 30 Japanese subjects viewing images of 40 common objects from a previous study. In Chapter 3, the notion of a unary classifier based on cosine similarity, which is a common measure of neural similarity, was introduced. A unary classifier is trained on the positive samples only and has the advantage of reducing false positives. The unary classification method increased the average area under the receiver operating characteristic curve (ROC AUC) from 0.801 (SD=0.138) in a state-of-the-art (SOTA) model to 0.812 (SD=0.135) for fMRI decoding in the ventral temporal cortex (VT). The unary decoder also achieved above-chance performance for the dorsolateral prefrontal cortex (DLPFC; ROC AUC=0.570, SD=0.113, t(77)=5.471, p<.001), which is known to be difficult to decode given its underlying physiological properties. To better understand the role of neural similarity in fMRI decoding, correlation analyses were conducted between the VT data of two different demographics, the original group from Japan (N=30) and another group from the US (N=28). The high correlation (r=0.960, p<.001) between the neural similarity patterns of the two groups suggests that neural similarity is a valid measure of decodability and semantic topology. Machine learning, especially deep learning, has significantly improved the SOTA models in many domains. In Chapter 4, the similarity in semantic topology between our brain and popular natural language processing (NLP) models was examined. A moderate correlation between VT and NLP was found (r=0.643, p<.001, N=780), which suggests that transfer learning from NLP models can augment fMRI decoding. Chapter 5 shows that by incorporating NLP models into the unary decoder, well-above-chance classification performance could be achieved on objects that were previously not present in the dataset. This thesis lays the foundation of unary fMRI decoding and transfer learning from SOTA machine learning models to improve DecNef performances in the future.