This work proposes a novel generative multimodal approach to jointly analyze multimodal data while linking the multimodal information to colors. We apply our proposed framework, which disentangles multimodal data into private and shared sets of features from pairs of structural (sMRI), functional (sFNC and ICA), and diffusion MRI data (FA maps). With our approach, we find that heterogeneity in schizophrenia is potentially a function of modality pairs. Results show (1) schizophrenia is highly multimodal and includes changes in specific networks, (2) non-linear relationships with schizophrenia are observed when interpolating among shared latent dimensions, and (3) we observe a decrease in the modularity of functional connectivity and decreased visual-sensorimotor connectivity for schizophrenia patients for the FA-sFNC and sMRI-sFNC modality pairs, respectively. Additionally, our results generally indicate decreased fractional corpus callosum anisotropy, and decreased spatial ICA map and voxel-based morphometry strength in the superior frontal lobe as found in the FA-sFNC, sMRI-FA, and sMRI-ICA modality pair clusters. In sum, we introduce a powerful new multimodal neuroimaging framework designed to provide a rich and intuitive understanding of the data which we hope challenges the reader to think differently about how modalities interact.

Characterizing neuropsychiatric disorders is challenging due to heterogeneity in the population. We propose combining structural and functional neuroimaging and genomic data in a multimodal classification framework to leverage their complementary information. Our objectives are two-fold (i) to improve the classification of disorders and (ii) to introspect the concepts learned to explore underlying neural and biological mechanisms linked to mental disorders. Previous multimodal studies have focused on naïve neural networks, mostly perceptron, to learn modality-wise features and often assume equal contribution from each modality. Our focus is on the development of neural networks for feature learning and implementing an adaptive control unit for the fusion phase. Our mid fusion with attention model includes a multilayer feed-forward network, an autoencoder, a bi-directional long short-term memory unit with attention as the features extractor, and a linear attention module for controlling modality-specific influence. The proposed model acquired 92% (p < .0001) accuracy in schizophrenia prediction, outperforming several other state-of-the-art models applied to unimodal or multimodal data. Post hoc feature analyses uncovered critical neural features and genes/biological pathways associated with schizophrenia. The proposed model effectively combines multimodal neuroimaging and genomics data for predicting mental disorders. Interpreting salient features identified by the model may advance our understanding of their underlying etiological mechanisms.

Electrochemical polarization, which often plays a critical role in driving chemical reactions at solid-liquid interfaces, can arise spontaneously through the exchange of ions and/or electrons across the interface. However, the extent to which such spontaneous polarization prevails at nonconductive interfaces remains unclear because such materials preclude measuring and controlling the degree of interfacial polarization via standard (i.e., wired) potentiometric methods. Herein, we circumvent the limitations of wired potentiometry by applying infrared and ambient pressure X-ray photoelectron spectroscopies (AP-XPS) to probe the electrochemical potential of nonconductive interfaces as a function of solution composition. As a model class of macroscopically nonconductive interfaces, we specifically probe the degree of spontaneous polarization of ZrO₂-supported Pt and Au nanoparticles immersed in aqueous solutions of varying pH. Shifts in the Pt-adsorbed CO vibrational band position evince electrochemical polarization of the Pt/ZrO₂-water interface with changing pH, and AP-XPS reveals quasi-Nernstian shifts of the electrochemical potential of Pt and Au with pH in the presence of H₂. These results indicate that spontaneous proton transfer via equilibrated H⁺/H₂ interconversion spontaneously polarizes metal nanoparticles even when supported on a nonconductive host. Consequently, these findings indicate that solution composition (i.e., pH) can be an effective handle for tuning interfacial electrical polarization and potential at nonconductive interfaces.

The analysis of time-varying activity and connectivity patterns (i.e., the chronnectome) using resting-state magnetic resonance imaging has become an important part of ongoing neuroscience discussions. The majority of previous work has focused on variations of temporal coupling among fixed spatial nodes or transition of the dominant activity/connectivity pattern over time. Here, we introduce an approach to capture spatial dynamics within functional domains (FDs), as well as temporal dynamics within and between FDs. The approach models the brain as a hierarchical functional architecture with different levels of granularity, where lower levels have higher functional homogeneity and less dynamic behavior and higher levels have less homogeneity and more dynamic behavior. First, a high-order spatial independent component analysis is used to approximate functional units. A functional unit is a pattern of regions with very similar functional activity over time. Next, functional units are used to construct FDs. Finally, functional modules (FMs) are calculated from FDs, providing an overall view of brain dynamics. Results highlight the spatial fluidity within FDs, including a broad spectrum of changes in regional associations, from strong coupling to complete decoupling. Moreover, FMs capture the dynamic interplay between FDs. Patients with schizophrenia show transient reductions in functional activity and state connectivity across several FDs, particularly the subcortical domain. Activity and connectivity differences convey unique information in many cases (e.g., the default mode) highlighting their complementarity information. The proposed hierarchical model to capture FD spatiotemporal variations provides new insight into the macroscale chronnectome and identifies changes hidden from existing approaches.

The brain is highly dynamic, reorganizing its activity at different interacting spatial and temporal scales, including variation within and between brain networks. The chronnectome is a model of the brain in which nodal activity and connectivity patterns change in fundamental and recurring ways over time. Most literature assumes fixed spatial nodes/networks, ignoring the possibility that spatial nodes/networks may vary in time. Here, we introduce an approach to calculate a spatially fluid chronnectome (called the spatial chronnectome for clarity), which focuses on the variations of networks coupling at the voxel level, and identify a novel set of spatially dynamic features. Results reveal transient spatially fluid interactions between intra- and internetwork relationships in which brain networks transiently merge and separate, emphasizing dynamic segregation and integration. Brain networks also exhibit distinct spatial patterns with unique temporal characteristics, potentially explaining a broad spectrum of inconsistencies in previous studies that assumed static networks. Moreover, we show anticorrelative connections to brain networks are transient as opposed to constant across the entire scan. Preliminary assessments using a multi-site dataset reveal the ability of the approach to obtain new information and nuanced alterations that remain undetected during static analysis. Patients with schizophrenia (SZ) display transient decreases in voxel-wise network coupling within visual and auditory networks, and higher intradomain coupling variability. In summary, the spatial chronnectome represents a new direction of research enabling the study of functional networks which are transient at the voxel level, and the identification of mechanisms for within- and between-subject spatial variability.