awesome-distribution-shift

A curated list of papers and resources about the distribution shift in machine learning. I categorize them based on their topic and content. I will try to make this list updated.

Here is an example of distribution shift in images across domains from DomainBed.

I categorize the papers on distribution shift as follows. If you found any error or any missed paper, please don't hesitate to add.

To be updated

Contents

• awesome-distribution-shift
  • Contents
  • Benchmark
  • Generalization
    • Domain Generalization & Out-of-distribution Robustness
      • Out-of-distribution Robustness
      • Domain Generalization
    • Domain Adaptation
    • Test-time Adaptation/Training
  • Data Modality
    • Graph
    • Text
    • Time Series
    • Video
    • Speech
    • Tabular Data
    • Others (RecSys)
  • Decentralized (Federated)
  • Detection
  • Fairness
  • Robustness
  • Learning Strategy

Benchmark

• [ICLR 2021] In Search of Lost Domain Generalization [paper] [code (DomainBed)]
• [ICLR 2021] BREEDS: Benchmarks for Subpopulation Shift [paper] [code]
• [ICML 2021] WILDS: A Benchmark of in-the-Wild Distribution Shifts [paper] [code]
• [NeurIPS 2021] Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks [paper] [code]
• [ICLR 2022] MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts [paper] [code]
• [NeurIPS 2022] GOOD: A Graph Out-of-Distribution Benchmark [paper] [code]
• [NeurIPS 2022] BOND: Benchmarking Unsupervised Outlier Node Detection on Static Attributed Graphs [paper] [code]
• [NeurIPS 2022] ADBench: Anomaly Detection Benchmark [paper] [code]
• [NeurIPS 2022] Wild-Time: A Benchmark of in-the-Wild Distribution Shifts over Time [paper] [code]
• [NeurIPS 2022] OpenOOD: Benchmarking Generalized Out-of-Distribution Detection [paper] [code]
• [NeurIPS 2022] AnoShift: A Distribution Shift Benchmark for Unsupervised Anomaly Detection [paper] [code]
• [CVPR 2022] OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization [paper] [code]
• [CVPR 2022] The Auto Arborist Dataset: A Large-Scale Benchmark for Multiview Urban Forest Monitoring Under Domain Shift [paper] [code]
• [ECCV 2022] OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images [paper] [code]
• [arxiv 2022] DrugOOD: OOD Dataset Curator and Benchmark for AI Aided Drug Discovery [paper] [code]

Generalization

Domain Generalization & Out-of-distribution Robustness

There are mainly two types of distribution shift: domain shift (testing on unseen domains) and subpopulation shift (the domains of testing data are seen but underrepresented in the training data). Below figure from GOOD well demonstrates them. Domain Generalization mainly studies domain shift, while Out-of-distribution Robustness studies both of them. These two research directions are very related and share a lot in common.

Out-of-distribution Robustness

• [arxiv 2019] Invariant Risk Minimization [paper]
• [ICLR 2020] Distributionally Robust Neural Networks for Group Shifts: on the Importance of Regularization for Worst-Case Generalization [paper]
• [ICLR 2021] In-N-Out: Pre-Training and Self-Training Using Auxiliary Information for Out-of-Distribution Robustness [paper]
• [ICML 2021] Out-of-Distribution Generalization Via Risk Extrapolation (REx) [paper]
• [ICML 2021] Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and in-Distribution Generalization [paper]
• [NeurIPS 2021] Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization [paper]
• [ICML 2022] Improving Out-of-Distribution Robustness Via Selective Augmentation [paper]
• [ICML 2022] Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization [paper]
• [NeurIPS 2022] Diverse Weight Averaging for Out-of-Distribution Generalization [paper]

W/o group label

• [ICML 2021] Just Train Twice: Improving Group Robustness Without Training Group Information [paper]
• [ICML 2022] Model Agnostic Sample Reweighting for Out-of-Distribution Learning [paper]
• [ICML 2022] Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations [paper]

Domain Generalization

• [AAAI 2018] Learning to Generalize: Meta-Learning for Domain Generalization [paper]
• [CVPR 2018] Domain Generalization With Adversarial Feature Learning [paper]
• [ECCV 2018] Deep Domain Generalization Via Conditional Invariant Adversarial Networks [paper]
• [NeurIPS 2018] MetaReg: Towards Domain Generalization Using Meta-Regularization [paper]
• [NeurIPS 2019] Domain Generalization Via Model-Agnostic Learning of Semantic Features [paper]
• [CVPR 2019] Episodic Training for Domain Generalization [paper]
• [CVPR 2019] Domain Generalization By Solving Jigsaw Puzzles [paper]
• [NeurIPS 2020] Domain Generalization Via Entropy Regularization [paper]
• [ECCV 2020] Learning to Optimize Domain Specific Normalization for Domain Generalization [paper]
• [ECCV 2020] Learning From Extrinsic and Intrinsic Supervisions for Domain Generalization [paper]
• [ECCV 2020] Learning to Balance Specificity and Invariance for in and Out of Domain Generalization [paper]
• [ECCV 2020] Self-Challenging Improves Cross-Domain Generalization [paper]
• [ECCV 2020] Learning to Generate Novel Domains for Domain Generalization [paper]
• [CVPR 2020] Learning to Learn Single Domain Generalization [paper]
• [JMLR 2021] Domain Generalization By Marginal Transfer Learning [paper]
• [NeurIPS 2021] Swad: Domain Generalization By Seeking Flat Minima [paper]
• [NeurIPS 2021] Model-based Domain Generalization [paper]
• [ICCV 2021] Selfreg: Self-Supervised Contrastive Regularization for Domain Generalization [paper]
• [ICCV 2021] A Simple Feature Augmentation for Domain Generalization [paper]
• [ICLR 2022] Gradient Matching for Domain Generalization [paper]
• [CVPR 2022] PCL: Proxy-Based Contrastive Learning for Domain Generalization [paper]
• [ECCV 2022] Domain Generalization By Mutual-Information Regularization With Pre-Trained Models [paper]
• [ICML 2022] Dna: Domain Generalization With Diversified Neural Averaging [paper]
• [NeurIPS 2022] Domain Generalization Without Excess Empirical Risk [paper]
• [ICCV 2023] PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization [paper]

W/o domain label

Explicitly introduce compound domain generalization without domain labels (some previous work also don't rely on domain labels)

• [CVPR 2022] Compound Domain Generalization Via Meta-Knowledge Encoding [paper]

Unsupervised

• [CVPR 2022] Towards Unsupervised Domain Generalization [paper]
• [CVPR 2022] Unsupervised Domain Generalization By Learning a Bridge Across Domains [paper]

Domain Adaptation

In the setting of domain adaptation, the model can access partial target domain data (usually unsupervised). There are also many variants like open-set, evolving, dynamic domain adaptation, etc.

• [ICML 2015] Unsupervised Domain Adaptation By Backpropagation [paper]
• [JMLR 2016] Domain-Adversarial Training of Neural Networks [paper]
• [ECCV 2016] Deep Coral: Correlation Alignment for Deep Domain Adaptation [paper]
• [NeurIPS 2016] Unsupervised Domain Adaptation With Residual Transfer Networks [paper]
• [NeurIPS 2016] Learning Transferrable Representations for Unsupervised Domain Adaptation [paper]
• [CVPR 2017] Adversarial Discriminative Domain Adaptation [paper]
• [ICDM 2017] Balanced Distribution Adaptation for Transfer Learning [paper]
• [ECCV 2018] Open set domain adaptation by backpropagation [paper]
• [AAAI 2018] Multi-Adversarial Domain Adaptation [paper]
• [MM 2018] Visual Domain Adaptation With Manifold Embedded Distribution Alignment [paper]
• [CVPR 2019] Universal Domain Adaptation [paper]
• [CVPR 2019] Contrastive Adaptation Network for Unsupervised Domain Adaptation [paper]
• [ICML 2019] On Learning Invariant Representations for Domain Adaptation [paper]
• [CVPR 2020] Reliable Weighted Optimal Transport for Unsupervised Domain Adaptation [paper]
• [ICML 2020] Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation [paper]
• [TPAMI 2020] Maximum Density Divergence for Domain Adaptation [paper]
• [AAAI 2020] Adversarial Domain Adaptation with Domain Mixup [paper]
• [WACV 2021] Dacs: Domain Adaptation Via Cross-Domain Mixed Sampling [paper]
• [CVPR 2021] Dynamic Weighted Learning for Unsupervised Domain Adaptation [paper]
• [ICML 2022] Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation [paper]
• [TPAMI 2022] Cross-domain Contrastive Learning for Unsupervised Domain Adaptation [paper]
• [WACV 2022] Federated Multi-target Domain Adaptation [paper]

Test-time Adaptation/Training

In the scenario of test-time adaptation, the model is pre-trained on the source domain data and is adapted during the testing on the target domain.

• [ICML 2020] Test-time training with self-supervision for generalization under distribution shifts [paper]
• [MICCAI 2020] Test-time Unsupervised Domain Adaptation [paper]
• [ICLR 2021] Tent: Fully test-time adaptation by entropy minimization [paper]
• [NeurIPS 2021] TTT++: When Does Self-supervised Test-time Training Fail or Thrive? [paper]
• [ICML 2022] Mt3: Meta Test-time Training for Self-supervised Test-time Adaption [paper]
• [CVPR 2022] Contrastive Test-Time Adaptation [paper]
• [CVPR 2022] Parameter-free Online Test-time Adaptation [paper]
• [NeurIPS 2022] Test-Time Adaptation via Conjugate Pseudo-labels [paper]
• [ICML 2022] Efficient Test-Time Model Adaptation without Forgetting [paper]
• [NeurIPS 2022] Test-time Training with Masked Autoencoders [paper]
• [NeurIPS 2022] Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts [paper]

Also known as source-free domain adaptation

• [CVPR 2020] Universal Source-Free Domain Adaptation [paper]
• [CVPR 2020] Model adaptation: Unsupervised domain adaptation without source data [paper]
• [ICCV 2021] Generalized Source-Free Domain Adaptation [paper]
• [ICCV 2021] Adaptive Adversarial Network for Source-Free Domain Adaptation [paper]
• [CVPR 2022] Source-Free Domain Adaptation via Distribution Estimation [paper]
• [ICML 2022] Balancing Discriminability and Transferability for Source-Free Domain Adaptation [paper]
• [NeurIPS 2022] Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation [paper]
• [NeurIPS 2022] Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning [paper]

Data Modality

Above papers study the distribution shift on images. There are also many applications to other data modalities.

Graph

Adaptation

• [TKDE 2022] Graph Transfer Learning via Adversarial Domain Adaptation with Graph Convolution [paper (2019)]
• [IJCAI 2019] DANE: Domain Adaptive Network Embedding [paper]
• [WWW 2020] Unsupervised Domain Adaptive Graph Convolutional Networks [paper]
• [ICLR 2022] Graph-Relational Domain Adaptation [paper]
• [AAAI 2023] Non-IID Transfer Learning on Graphs [paper]
• [ICLR 2023] Graph Domain Adaptation via Theory-Grounded Spectral Regularization [paper]

Generalization

• [ICLR 2022] Handling Distribution Shifts on Graphs: An Invariance Perspective [paper]
• [ICLR 2022] Discovering Invariant Rationales for Graph Neural Networks [paper]
• [ICML 2022] Interpretable and Generalizable Graph Learning Via Stochastic Attention Mechanism [paper]
• [KDD 2022] Causal Attention for Interpretable and Generalizable Graph Classification [paper]
• [NeurIPS 2022] Dynamic Graph Neural Networks Under Spatio-temporal Distribution Shift [paper]
• [NeurIPS 2022] Learning Substructure Invariance for Out-of-Distribution Molecular Representations [paper]
• [NeurIPS 2022] Learning Invariant Graph Representations for Out-of-Distribution Generalization [paper]
• [NeurIPS 2022] Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs [paper]
• [WSDM 2023] Alleviating Structural Distribution Shift in Graph Anomaly Detection [paper]

Test-time adaptation

• [arxiv 2022] Test-Time Training for Graph Neural Networks [paper]
• [arxiv 2022] Empowering Graph Representation Learning with Test-time Graph Transformation [paper]

Text

Many applications in different NLP tasks.

• [CVPR 2019] Sequence-to-sequence Domain Adaptation Network for Robust Text Image Recognition [paper]
• [NAACL 2019] Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation [paper]
• [AAAI 2020] Multi-source Domain Adaptation for Text Classification Via Distancenet-bandits [paper]
• [COLING 2020] Semi-supervised Domain Adaptation for Dependency Parsing via Improved Contextualized Word Representations [paper]
• [ACL 2020] Pretrained Transformers Improve Out-of-distribution Robustness [paper]
• [EMNLP 2021] Contrastive Domain Adaptation for Question Answering Using Limited Text Corpora [paper]
• [EMNLP 2021] Pdaln: Progressive Domain Adaptation over a Pre-trained Model for Low-resource Cross-domain Named Entity Recognition [paper]
• [ACL 2021] Matching Distributions Between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation [paper]
• [ACL 2021] Bridge-based Active Domain Adaptation for Aspect Term Extraction [paper]
• [ECCV 2022] Grounding Visual Representations with Texts for Domain Generalization [paper]
• [KAIS 2022] Knowledge distillation for bert unsupervised domain adaptation [paper]
• [ACL 2022] Semi-supervised Domain Adaptation for Dependency Parsing with Dynamic Matching Network [paper] ...

Time Series

• [ICLR 2017] Variational Recurrent Adversarial Deep Domain Adaptation [paper]
• [KDD 2020] Multi-Source Deep Domain Adaptation with Weak Supervision for Time-Series Sensor Data [paper
• [AAAI 2021] Time Series Domain Adaptation Via Sparse Associative Structure Alignment [paper]
• [IJCAI 2021] Adversarial Spectral Kernel Matching for Unsupervised Time Series Domain Adaptation [paper]
• [CHIL 2021] An Empirical Framework for Domain Generalization in Clinical Settings [paper]
• [TNNLS 2022] Self-Supervised Autoregressive Domain Adaptation for Time Series Data [paper]
• [MM 2022] Domain Adaptation for Time-series Classification to Mitigate Covariate Shift [paper]
• [ICRA 2022] Causal-based Time Series Domain Generalization for Vehicle Intention Prediction [paper]
• [ICML 2022] Domain Adaptation for Time Series Forecasting via Attention Sharing [paper]

Video

• [ICCV 2019] Temporal Attentive Alignment for Large-scale Video Domain Adaptation [paper]
• [ECCV 2020] Shuffle and Attend: Video Domain Adaptation [paper]
• [ICCV 2021] Learning Cross-modal Contrastive Features for Video Domain Adaptation [paper]
• [NeurIPS 2021] Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing [paper]
• [ECCV 2022] Source-free Video Domain Adaptation by Learning Temporal Consistency for Action Recognition [paper]
• [WACV 2022] Domain Generalization Through Audio-visual Relative Norm Alignment in First Person Action Recognition [paper]

Speech

• [SPL 2014] Autoencoder-based Unsupervised Domain Adaptation for Speech Emotion Recognition [paper]
• [ICASSP 2015] Supervised Domain Adaptation for Emotion Recognition from Speech [paper]
• [NeuroComputing 2017] An Unsupervised Deep Domain Adaptation Approach for Robust Speech Recognition [paper]
• [ASRU 2019] Domain Adaptation Via Teacher-student Learning for End-to-end Speech Recognition [paper]
• [TAC 2021] Improving Cross-corpus Speech Emotion Recognition with Adversarial Discriminative Domain Generalization (addog) [paper]
• [SLT 2021] Domain Generalization with Triplet Network for Cross-corpus Speech Emotion Recognition [paper]
• [ICASSP 2022] Large-scale Asr Domain Adaptation Using Self-and Semi-supervised Learning [paper]

Tabular Data

• [NeurIPS 2022] Distribution-Informed Neural Networks for Domain Adaptation Regression [paper]
• [NeurIPS 2022] C-Mixup: Improving Generalization in Regression [paper]

Others (RecSys)

• [KDD 2021] Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System [paper]
• [WWW 2021] Disentangling User Interest and Conformity for Recommendation with Causal Embedding [paper]
• [SIGIR 2021] Causal Intervention for Leveraging Popularity Bias in Recommendation [paper]
• [SIGIR 2021] Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue [paper]
• [SIGIR 2021] AutoDebias: Learning to Debias for Recommendation [paper]
• [WWW 2022] Cross Pairwise Ranking for Unbiased Item Recommendation [paper]
• [WWW 2022] Causal Representation Learning for Out-of-Distribution Recommendation [paper]
• [SIGIR 2022] Interpolative Distillation for Unifying Biased and Debiased Recommendation [paper]
• [TOIS 2022] Addressing Confounding Feature Issue for Causal Recommendation [paper]
• [TKDE 2023] Popularity Bias Is Not Always Evil: Disentangling Benign and Harmful Bias for Recommendation [paper]

...

Decentralized (Federated)

Besides generalization of centralized learning, transferability of decentralized setting (Federated Learning) has also received attention.

• [arxiv 2018] Federated Learning with Non-IID Data [paper]
• [ICML 2019] Agnostic Federated Learning [paper]
• [MLSys 2020] Federated Optimization in Heterogeneous Networks [paper]
• [ICML 2020] SCAFFOLD: Stochastic Controlled Averaging for Federated Learning [paper]
• [NeurIPS 2020] Robust Federated Learning: the Case of Affine Distribution Shifts [paper]
• [ICLR 2021] Fedbn: Federated Learning on Non-iid Features Via Local Batch Normalization [paper
• [SDM 2021] Fairness-aware Agnostic Federated Learning [paper]
• [CVPR 2021] FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space [paper]
• [NeurIPS 2022] FedSR: A Simple and Effective Domain Generalization Method for Federated Learning [paper]
• [TPDS 2022] Flexible Clustered Federated Learning for Client-level Data Distribution Shift [paper]
• [CVPR 2022] Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning [paper]
• [CVPR 2022] Feddc: Federated Learning with Non-iid Data Via Local Drift Decoupling and Correction [paper]

Detection

Besides generalization, other perspectives including detection, fairness robustness, etc. are also studied.

Out-of-distribution detection, outlier detection and anomaly detection.

• [ICDM 2018] Adversarially Learned Anomaly Detection [paper]
• [arxiv 2018] Learning Confidence for Out-of-distribution Detection in Neural Networks [paper]
• [NeurIPS 2019] Likelihood Ratios for Out-of-distribution Detection [paper]
• [IJCNN 2019] XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning [paper]
• [ICDM 2020] Copod: Copula-based Outlier Detection [paper]
• [NeurIPS 2020] Energy-based Out-of-distribution Detection [paper]
• [CVPR 2020] Learning Memory-guided Normality for Anomaly Detection [paper]
• [NeurIPS 2021] Exploring the Limits of Out-of-distribution Detection [paper]
• [NeurIPS 2021] Automatic Unsupervised Outlier Model Selection [paper]
• [ICCV 2021] Divide-and-assemble: Learning Block-wise Memory for Unsupervised Anomaly Detection [paper]
• [CVPR 2021] Anomaly Detection in Video Via Self-supervised and Multi-task Learning [paper]
• [AAAI 2022] On the Impact of Spurious Correlation for Out-of-distribution Detection [paper]
• [AAAI 2022] Lunar: Unifying Local Outlier Detection Methods Via Graph Neural Networks [paper]
• [ICML 2022] Poem: Out-of-distribution Detection with Posterior Sampling [paper]
• [ECCV 2022] Dice: Leveraging Sparsification for Out-of-distribution Detection [paper]

Fairness

• [FAccT 2021] Fairness Violations and Mitigation Under Covariate Shift [paper]
• [AAAI 2021] Robust Fairness Under Covariate Shift [paper]
• [NeurIPS 2022] Diagnosing Failures of Fairness Transfer Across Distribution Shift in Real-world Medical Settings [paper]
• [NeurIPS 2022] Fairness Transferability Subject to Bounded Distribution Shift [paper]
• [NeurIPS 2022] Transferring Fairness under Distribution Shifts via Fair Consistency Regularization [paper]
• [arxiv 2022] How Robust is Your Fairness? Evaluating and Sustaining Fairness under Unseen Distribution Shifts [paper]

Robustness

• [ICLR 2019] On the Sensitivity of Adversarial Robustness to Input Data Distributions [paper]
• [CVPR 2021] Adversarial Robustness under Long-Tailed Distribution [paper]
• [arxiv 2022] BOBA: Byzantine-Robust Federated Learning with Label Skewness [paper]
• [arxiv 2022] Generalizability of Adversarial Robustness Under Distribution Shifts [paper]

Learning Strategy

• [ICLR 2020] Learning to Balance: Bayesian Meta-learning for Imbalanced and Out-of-distribution Tasks [paper]
• [NeurIPS 2020] OOD-MAML: Meta-learning for few-shot out-of-distribution detection and classification [paper]
• [NeurIPS 2020] Task-robust Model-agnostic Meta-learning [paper]
• [TIP 2021] Domain Adaptive Ensemble Learning [paper]
• [NeurIPS 2021] Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution [paper]
• [ICLR 2022] Deep Ensembling with No Overhead for either Training or Testing: The All-Round B- lessings of Dynamic Sparsity [paper]
• [NeurIPS 2022] Improving Multi-Task Generalization via Regularizing Spurious Correlation [paper]

...