You can also run this tutorial interactively on Google Colab!
NOTE: Besides the Algonauts Project 2023 Challenge, you might still find this tutorial useful as an introduction to fMRI data and encoding models of the visual brain.
The quest to understand the nature of human intelligence and engineer more advanced forms of artificial intelligence are becoming increasingly intertwined. The Algonauts Project brings biological and machine intelligence researchers together on a common platform to exchange ideas and advance both fields. The 2023 challenge focuses on explaining neural visual responses to complex naturalistic scenes.
The Challenge is based on the Natural Scenes Dataset (NSD), a massive dataset of 7T fMRI responses to images of natural scenes coming from the COCO dataset. The goal of the Challenge is to promote the development of cutting-edge encoding models which accurately predict (i.e., encode) the fMRI responses to visual input. You can read more about the Challenge in the website and paper.
Here we provide the development kit tutorial of the Challenge (algonauts_2023_challenge_tutorial.ipynb) where you will learn how to:
- Load and visualize the Challenge fMRI data, functionally different areas of the visual cortex called regions-of-interest (ROIs), and the stimulus images.
- Build linearizing encoding models using a pretrained AlexNet architecture, evaluate them and visualize the resulting prediction accuracy (i.e., encoding accuracy).
- Prepare the predicted brain responses to the test images in the right format for submission to the Challenge leaderboard.
Please watch this video for an introduction to the Algonauts Project 2023 Challenge and a walkthrough of this development kit tutorial.
In the code folder you will code used for the Challenge, split into two subfolders:
01_challenge_data_preparation: Code used to extract the challenge data from NSD, and prepare it in Challenge format.02_baseline_encoding_model: Code used to train and test the Challenge baseline encoding model.
NOTE: Even if you're not interested in the Algonauts Project 2023 Challenge, you might still find this tutorial useful as an introduction to fMRI data and encoding models of the visual brain.
Should you experience problems with the code, please get in touch with Ale (alessandro.gifford@gmail.com).
Challenge DevKit Tutorial (Colab)
Challenge DevKit Tutorial (GitHub)
Challenge Overview and Colab Tutorial Walkthrough Video
CodaLab Challenge Submission Page
CodaLab Submission Walkthrough Video
Adapt COCO Metadata to Cropped NSD Images
If you use the data provided for the Algonauts Project 2023 Challenge please cite the following papers:
-
Gifford AT, Lahner B, Saba-Sadiya S, Vilas MG, Lascelles A, Oliva A, Kay K, Roig G, Cichy RM. 2023. The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes. arXiv preprint, arXiv:2301.03198. DOI: https://doi.org/10.48550/arXiv.2301.03198
-
Allen EJ, St-Yves G, Wu Y, Breedlove JL, Prince JS, Dowdle LT, Nau M, Caron B, Pestilli F, Charest I, Hutchinson JB, Naselaris T, Kay K. 2022. A massive 7T fMRI dataset to bridge cognitive neuroscience and computational intelligence. Nature Neuroscience, 25(1):116–126. DOI: https://doi.org/10.1038/s41593-021-00962-x