Codes of Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
- Linux machine with four GPUs
Conda
- Python's dependencies: this file:
conda create --name research --file spec-file.txt
conda activate research
git clone git@github.com:NVIDIA/apex.git
cd apex
git checkout 4ef930c1c884fdca5f472ab2ce7cb9b505d26c1a # to fix the exact library version
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch
In NLP experiments, we need similar words constructed from fasttext's pre-trained word embeddings.
cd code
wget https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip
python nlp_construct_simiar_data.py
Please fill the value of replace_data
in conf/dataset/ag_news.yaml
with the generated file's path. For example,
replace_data: /home/nzw/code/ag_replace_ids.npy
Please run content of all scripts in ./scripts/**/train/
under code
.
After training, please run ./gather_weights.py
to generate text files for evaluation.
Please run content of all scripts in ./scripts/**/eval/
under code
as well.
For the AG news dataset, please run code/notebooks/filter_ag_news.ipynb
after evaluation of mean classifier before the other evaluation scripts such as linear classifier and bound computation.
To obtain all figures and tables in the paper, you run notebooks in code/notebooks/
. The codes save generated figures and tables into ./doc/figs
and ./doc/tabs
, respectively.
code/notebooks/bound.ipynb
generates Figure 1, the content of Table 2, and Tables 4 and 5.code/notebooks/coupon.ipynb
generates Table 3.code/notebooks/collision.ipynb
computes upper bound of the collision term used incode/notebooks/bound.ipynb
.code/notebooks/collosion_analysis.ipynb
generates Figures 3, 4, and 5.
@inproceedings{NS2021,
title = {Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning},
author = {Kento Nozawa, Issei Sato},
year = {2021},
booktitle = {NeurIPS},
pages = {5784--5797},
}