Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

https://arxiv.org/abs/2110.09246

This is a fast and scalable approach for detecting out-of-distribution test samples. It can be applied to any pretrained model.

Pseudocode

# Assuming to have: trainloader, testloader, and model with model.backbone() and model.classifer() methods

# Extract the training set features. Dimensions 2D matrix: training_size x num_features
features = torch.vstack([model.backbone(images) for images, label in trainloader])
norm = torch.linalg.norm(features, dim=-1, keepdim=True)
features = features / norm

# Compute the training data empirical correlation matrix inverse
x_t_x_inv = torch.linalg.inv(features.T @ features)

# Calculate the regret: Large regret means out-of-distribution sample
for images, labels in testloader:
    features = model.backbone(images)

    # Normalize
    norm = torch.linalg.norm(features, dim=-1, keepdim=True)
    features = features / norm
    
    # Get the probabilities of the normalized features
    probs = torch.softmax(model.classifier(features), dim=-1)

    # Calc projection
    x_proj = features @ x_t_x_inv @ features.T
    xt_g = x_proj / (1 + x_proj)

    # Equation 20
    n_classes = probs.shape[-1]
    nf = torch.sum(probs / (probs + (1 - probs) * (probs ** x_t_g)), dim=-1)
    regrets = torch.log(nf) / torch.log(torch.tensor(n_classes))

Paper results

Run to code

Install requirements

# Create env
conda create -n pnml_ood python=3.8.0 --yes
conda activate pnml_ood

# Install pip for fallback
conda install --yes pip

# Pytorch with GPU
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=10.2 -c pytorch --yes

# All other: Install with conda. If package installation fails, install with pip.
while read requirement; do conda install --yes $requirement || pip install $requirement; done < requirements.txt 

Download data and models

# Download OOD data
cd bash_scripts
chmod 777 ./download_data.sh
./download_data.sh

# Download pretrained models
chmod 777 ./download_models.sh
./download_models.sh

Optional: Download imagenet30

Follow https://github.com/alinlab/CSI

Imagenet30 training set: https://drive.google.com/file/d/1B5c39Fc3haOPzlehzmpTLz6xLtGyKEy4/view

Imagenet30 testing set: https://drive.google.com/file/d/13xzVuQMEhSnBRZr-YaaO08coLU2dxAUq/view

Put and untar under ./data/Imagenet30

.
├── README.md
├── data
│   ├── Imagenet30
│   │   ├── one_class_test
│   │   ├── one_class_test.tar
│   │   ├── one_class_train
│   │   └── one_class_train.tar

Execute methods

Execute a single method. Our pNML method runs on-top of the executed method

cd src
python main_execute_method.py model=densenet trainset=cifar100 method=baseline

Execute all methods

cd bash_scripts
chmod 777 ./execute_methods.sh
./execute_methods.sh

Create paper's tables

cd src
python main_create_tables.py

Citing

If you use this code in your research or wish to refer to the baseline results, please use the following BibTeX entry.

@inproceedings{bibas2021single,
  title={Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection},
  author={Bibas, Koby and Feder, Meir and Hassner, Tal},
  booktitle={Advances in Neural Information Processing Systems},
  year={2021}
}