Skip to content

Commit

Permalink
added instructions on how to use our code to replicate the ImageNet c…
Browse files Browse the repository at this point in the history
…lassification expts
  • Loading branch information
tyler-hayes committed Jun 25, 2020
1 parent 53acdbb commit 9c91167
Show file tree
Hide file tree
Showing 2 changed files with 78 additions and 15 deletions.
91 changes: 77 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,32 @@
REMIND
REMIND Your Neural Network to Prevent Catastrophic Forgetting
=====================================
This is a PyTorch implementation of the REMIND algorithm from our paper. An [arXiv pre-print](https://arxiv.org/abs/1910.02509) of our paper is available.

REMIND (REplay using Memory INDexing) is a novel brain-inspired streaming learning model that uses tensor quantization to efficiently store hidden representations (e.g., CNN feature maps) for later replay. REMIND implements this compression using Product Quantization (PQ) and outperforms existing models on the ImageNet and CORe50 classification datasets. Further, we demonstrate REMIND's robustness by pioneering streaming Visual Question Answering (VQA), in which an agent must answer questions about images.

Formally, REMIND takes an input image and passes it through frozen layers of a network to obtain tensor representations (feature maps). It then quantizes the tensors via PQ and stores the indices in memory for replay. The decoder reconstructs a previous subset of tensors from stored indices to train the plastic layers of the network before inference. We restrict the size of REMIND's replay buffer and use a uniform random storage policy.

![REMIND](./repo_images/REMIND_model.png)

## Dependencies
- Tested with Python 3.7.6 and PyTorch 1.3.1, torchvision 0.4.2, NumPy, FAISS, NVIDIA GPU
- Python >= 3.7.7
- PyTorch >= 1.3.1
- torchvision >= 0.4.2
- FAISS (installation instructions below)
- NumPy
- NVIDIA GPU

## Installing FAISS
We use the Product Quantization (PQ) implementation from the FAISS library. [See link.](https://github.com/facebookresearch/faiss/blob/master/INSTALL.md)
#### CPU version only
```conda install faiss-cpu -c pytorch```

#### GPU version
```
conda install faiss-gpu cudatoolkit=8.0 -c pytorch # For CUDA8
conda install faiss-gpu cudatoolkit=9.0 -c pytorch # For CUDA9
conda install faiss-gpu cudatoolkit=10.0 -c pytorch # For CUDA10
```

## Setup ImageNet-2012
The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) dataset has 1000 categories and 1.2 million images. The images do not need to be preprocessed or packaged in any database, but the validation images need to be moved into appropriate subfolders. [See link.](https://github.com/facebook/fb.resnet.torch/blob/master/INSTALL.md#download-the-imagenet-dataset)
Expand All @@ -23,18 +46,58 @@ The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) dataset has 1000
mkdir val && mv ILSVRC2012_img_val.tar val/ && cd val && tar -xvf ILSVRC2012_img_val.tar
wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash
```

## Installing FAISS
[See link.](https://github.com/facebookresearch/faiss/blob/master/INSTALL.md)
# CPU version only
```conda install faiss-cpu -c pytorch```

# GPU version
```
conda install faiss-gpu cudatoolkit=8.0 -c pytorch # For CUDA8
conda install faiss-gpu cudatoolkit=9.0 -c pytorch # For CUDA9
conda install faiss-gpu cudatoolkit=10.0 -c pytorch # For CUDA10
```
## Repo Structure & Descriptions

* [image_classification_experiments](./image_classification_experiments): files to replicate image classification experiments
* [imagenet_files](./image_classification_experiments/imagenet_files): files needed to run ImageNet experiments
* [imagenet_indices](./image_classification_experiments/imagenet_files/imagenet_indices): contains numpy files of labels for our ImageNet ordering
* [imagenet_train_labels.npy](./image_classification_experiments/imagenet_files/imagenet_indices/imagenet_train_labels.npy): numpy array with train labels for our ordering (can be generated for different order)
* [imagenet_val_labels.npy](./image_classification_experiments/imagenet_files/imagenet_indices/imagenet_val_labels.npy): numpy array with val labels for our ordering (can be generated for different order)
* [best_ResNet18ClassifyAfterLayer4_1_100.pth](./image_classification_experiments/imagenet_files/best_ResNet18ClassifyAfterLayer4_1_100.pth): our base init ckpt file trained on 100 fixed random classes of ImageNet (can be generated for different order)
* [imagenet_class_order.txt](./image_classification_experiments/imagenet_files/imagenet_class_order.txt): text file containing names of classes in order for training (supply this file for different order)
* [imagenet_base_initialization.py](./image_classification_experiments/imagenet_base_initialization.py): script to train an offline network on the base initialization data for ImageNet
* [imagenet_experiment.py](./image_classification_experiments/imagenet_experiment.py): script to train the streaming REMIND model on ImageNet
* [make_numpy_imagenet_label_files.py](./image_classification_experiments/make_numpy_imagenet_label_files.py): script to make `imagenet_train_labels.npy` and `imagenet_val_labels.npy` for particular ImageNet ordering
* [REMINDModel.py](./image_classification_experiments/REMINDModel.py): class for REMIND model
* [resnet_models.py](./image_classification_experiments/resnet_models.py): file containing variations of ResNet-18 architectures used in our experiments
* [retrieve_any_layer.py](./image_classification_experiments/retrieve_any_layer.py): script to extract features from different layers of a model
* [run_imagenet_experiment.sh](./image_classification_experiments/run_imagenet_experiment.sh): bash script to run the streaming REMIND model on ImageNet
* [train_base_init_network.sh](./image_classification_experiments/train_base_init_network.sh): bash script to train the base init model on a subset of ImageNet
* [train_base_init_network_from_scratch.py](./image_classification_experiments/train_base_init_network_from_scratch.py): script to train the base init model for ImageNet
* [utils.py](./image_classification_experiments/utils.py): overall utilities
* [utils_imagenet.py](./image_classification_experiments/utils_imagenet.py): ImageNet specific utilities

## Training REMIND on ImageNet (Classification)
We have provided the necessary files to train REMIND on the exact same ImageNet ordering used in our paper (provided in `imagenet_class_order.txt`). We also provide steps for running REMIND on an alternative ordering.

##### To train REMIND on the ImageNet ordering from our paper, follow the steps below:

1. Run `run_imagenet_experiment.sh` to train REMIND on the ordering from our paper. Note, this will use our ordering and associated files provided in `imagenet_files`.

##### To train REMIND on a different ImageNet ordering, follow the steps below:

1. Generate a text file containing one class name per line in the desired order.
2. Run `make_numpy_imagenet_label_files.py` to generate the necessary numpy files for the desired ordering using the text file from step 1.
3. Run `train_base_init_network.sh` to train an offline model using the desired ordering and label files generated in step 2 on the base init data.
4. Run `run_imagenet_experiment.sh` using the label files from step 2 and the ckpt file from step 3 to train REMIND on the desired ordering.

##### Files generated from the streaming experiment:

- `*.json` files containing incremental top-1 and top-5 accuracies
- `*.pth` files containing incremental model predictions/probabilities
- `*.pth` files containing incremental REMIND classifier (F) weights
- `*.pkl` files containing PQ centroids and incremental buffer data (e.g., latent codes)

##### To continue training REMIND from a previous ckpt:
We save out incremental weights and associated data for REMIND after each evaluation cycle. This enables REMIND to continue training from these saved files (in case of a computer crash etc.). This can be done as follows in `run_imagenet_experiment.sh`:

1. Set the `--resume_full_path` argument to the path where the previous REMIND model was saved.
2. Set the `--streaming_min_class` argument to the class REMIND left off on.
3. Run `run_imagenet_experiment.sh`

## Training REMIND on VQA Datasets
#### Coming Soon!

## Citation
If using this code, please cite our paper.
Expand All @@ -45,4 +108,4 @@ If using this code, please cite our paper.
journal={arXiv preprint arXiv:1910.02509},
year={2019}
}
```
2 changes: 1 addition & 1 deletion image_classification_experiments/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ def save_predictions(y_pred, min_class_trained, max_class_trained, save_path, su
max_class_trained)
else:
name = 'preds_min_trained_' + str(min_class_trained) + '_max_trained_' + str(max_class_trained) + suffix
torch.save(y_pred, save_path + '/' + name + '.pt')
torch.save(y_pred, save_path + '/' + name + '.pth')


def save_accuracies(accuracies, min_class_trained, max_class_trained, save_path, suffix='', order=None):
Expand Down

0 comments on commit 9c91167

Please sign in to comment.