SYDE Graduate Seminar - Learning Visual and Barrel Cortex Representations from Multimodal Convergence
Presenter
Candace Ng, MASc candidate in Systems Design Engineering
Abstract
Multisensory perception emerges from interactions between sensory pathways, but biologically inspired computational models often examine sensory modalities in isolation. This thesis presents a computational model of mouse visual and tactile processing through whiskers that incorporates anatomically motivated cortical pathways and modulation between sensory modalities. The visual pathway was based on MouseNet, a convolutional neural network derived from the organization of the mouse visual cortex. The whisker pathway was designed to reflect the hierarchical and temporal processing of signals through the somatosensory cortex and higher cortical regions. Representations produced from whisker input were used to modulate activity in the superficial layers of the primary visual cortex through a learned inhibitory gate motivated by experimentally observed somatosensory influences on visual processing. The visual and whisker encoders were trained jointly using a contrastive learning objective that aligned representations from paired simulated sensory observations. Training produced substantially greater similarity and retrieval performance for matched visual and whisker samples than for untrained representations, indicating that the model learned a shared embedding space across modalities. Representational similarity analysis was then used to compare model representations with neural recordings from the Allen Brain Observatory visual dataset and a dataset collected during object classification using whiskers. The learned visual representations exhibited measurable correspondence with recorded visual cortical activity, and inhibition from the whisker pathway produced a modest improvement in neural alignment under the evaluated conditions. Orientation selectivity increased in most evaluated visual layers despite not being explicitly included in the training objective, and trained VISp values were broadly comparable to physiological measurements from mouse visual cortex. Whisker representations retained condition-dependent structure but did not show consistent positive correspondence with the neural geometry, with animal-level RSA varying in sign and magnitude. These results demonstrate a framework for integrating visual and tactile processing within a biologically grounded neural network and evaluating the resulting representations directly against experimental neural data. This work provides a foundation for investigating how interactions between sensory modalities and anatomical constraints shape cortical representations.
This seminar counts towards the graduate student seminar attendance milestone!