This repository contains code for the DeepNSD project, an attempt to characterize the representational structure of human visual cortex with the massive NSD fMRI dataset and a bountiful cornucopia of deep neural network models. It also contains all source data and code for reproducing the results in the article: "A large-scale examination of inductive biases shaping high-level visual representation in brains and machines", previously pre-printed as "What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?".
Our Google Colab tutorial (bit.ly/Deep-NSD-Tutorial) provides a step by step demonstration of the main functions in this pipeline, fitting the representations of a CLIP model to a single subject subset of the fMRI data using the DeepDive package (soon to be re-released as DeepJuice).
You can use this codebase to quickly load (in a unified API) a number of models and their associated transforms. (Please note that -- pending further development -- you will have to install the underlying model packages manually, as they often require machine-specific settings during installation.)
Models we've preprocessed include:
- the PyTorch-Image-Models library
- the Torchvision model zoo
- the Taskonomy project
- the VISSL (SSL) model zoo
- ISL's MiDas models zoo
- FaceBook's DINO models...
(Note, the neural_data is included as a legacy folder, to preserve compatibility with older versions of the Colaboratory tutorial.)
To cite this repository, or the associated article, please use the following BibTex:
@article{conwell2024large,
title={A large-scale examination of inductive biases shaping high-level visual representation in brains and machines},
author={Conwell, Colin and Prince, Jacob S and Kay, Kendrick N and Alvarez, George A and Konkle, Talia},
journal={Nature Communications},
volume={15},
number={1},
pages={9383},
year={2024},
publisher={Nature Publishing Group UK London}
}- Squeezing your deep nets for science!
Recently, our team has been working on a new, highly-accelerated version of this codebase called Deepjuice -- effectively, a bottom-up reimplementation of all DeepDive functionalities that allows for end-to-end benchmarking (feature extraction, SRP, PCA, CKA, RSA, and regression) without ever removing data from the GPU.
DeepJuice is currently in private beta, but if you're interested in trying out, please feel free to contact me (Colin Conwell) by email: conwell[at]g[dot]harvard[dot]edu