More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activity

In the past few years, significant advancements were made in reconstruction\nof observed natural images from fMRI brain recordings using deep-learning\ntools. Here, for the first time, we show that dense 3D depth maps of observed\n2D natural images can also be recovered directly from fMRI brain recordings. We\nuse an off-the-shelf method to estimate the unknown depth maps of natural\nimages. This is applied to both: (i) the small number of images presented to\nsubjects in an fMRI scanner (images for which we have fMRI recordings -\nreferred to as "paired" data), and (ii) a very large number of natural images\nwith no fMRI recordings ("unpaired data"). The estimated depth maps are then\nused as an auxiliary reconstruction criterion to train for depth reconstruction\ndirectly from fMRI. We propose two main approaches: Depth-only recovery and\njoint image-depth RGBD recovery. Because the number of available "paired"\ntraining data (images with fMRI) is small, we enrich the training data via\nself-supervised cycle-consistent training on many "unpaired" data (natural\nimages & depth maps without fMRI). This is achieved using our newly defined and\ntrained Depth-based Perceptual Similarity metric as a reconstruction criterion.\nWe show that predicting the depth map directly from fMRI outperforms its\nindirect sequential recovery from the reconstructed images. We further show\nthat activations from early cortical visual areas dominate our depth\nreconstruction results, and propose means to characterize fMRI voxels by their\ndegree of depth-information tuning. This work adds an important layer of\ndecoded information, extending the current envelope of visual brain decoding\ncapabilities.\n