RMT4ML

This repository contains MATLAB and Python codes for visualizing random matrix theory results and their applications to machine learning, in Random Matrix Theory for Machine Learning.

In each subfolder (named after the corresponding section) there are:

• a .html file containing the MATLAB or IPython Notebook demos

• a .m or .ipynb source file

• Chapter 1 Introduction

  • Section 1.1 Motivation: The Pitfalls of Large-Dimensional Statistics
  • Section 1.2 Random Matrix Theory as an Answer
  • Section 1.3 Outline and Online Toolbox

• Chapter 2 Random Matrix Theory

  • Section 2.1 Fundamental Objects
  • Section 2.2 Foundational Random Matrix Results
    • Section 2.2.1 Key Lemma and Identities: Matlab code and Python code
    • Section 2.2.2 The Marcenko-Pastur and Semicircle Laws: Matlab code and Python code
    • Section 2.2.3 Large-Dimensional Sample Covariance Matrices and Generalized Semicircles: Matlab code and Python code
  • Section 2.3 Advanced Spectrum Considerations for Sample Covariances: Matlab code and Python code
  • Section 2.4 Preliminaries on Statistical Inference
    • Section 2.4.1 Linear Eigenvalue Statistics: Matlab code and Python code
    • Section 2.4.2 Eigenvector Projections and Subspace Methods: Matlab code and Python code
  • Section 2.5 Spiked Models: Matlab code and Python code
  • Section 2.6 Information-plus-Noise, Deformed Wigner, and Other Models
  • Section 2.7 Beyond Vectors of Independent Entries: Concentration of Measure in RMT
  • Section 2.8 Concluding Remarks
  • Section 2.9 Exercises

• Chapter 3 Statistical Inference in Linear Models

  • Section 3.1 Detection and Estimation in Information-plus-Noise Models
    • Section 3.1.1 GLRT Asymptotics: Matlab code and Python code
    • Section 3.1.2 Linear and Quadratic Discriminant Analysis: Matlab code and Python code
    • Section 3.1.1 Subspace Methods: The G-MUSIC Algorithm: Matlab code and Python code
  • Section 3.2 Covariance Matrix Distance Estimation: Matlab code and Python code
  • Section 3.3 M-Estimators of Scatter: Matlab code and Python code
  • Section 3.4 Concluding Remarks
  • Section 3.5 Practical Course Material:
    • The Wasserstein distance estimation: Matlab code and Python code
    • Robust portfolio optimization via Tyler estimator: Matlab code and Python code

• Chapter 4 Kernel Methods

  • Section 4.1 Basic Setting
  • Section 4.2 Distance and Inner-Product Random Kernel Matrices
    • Section 4.2.1 Main Intuitions
    • Section 4.2.2 Main Results: Distance Random Kernel Matrices: Matlab code and Python code
    • Section 4.2.3 Motivation: $\alpha-\beta$ Random Kernel Matrices
    • Section 4.2.4 Main Results: $\alpha-\beta$ Random Kernel Matrices: Matlab code and Python code
  • Section 4.3 Properly Scaling Kernel Model: Matlab code and Python code
  • Section 4.4 Implications to Kernel Methods
    • Section 4.4.1 Application to Kernel Spectral Clustering: Matlab code and Python code
    • Section 4.4.2 Application to Semi-supervised Kernel Learning: Matlab code and Python code
    • Section 4.4.3 Application to Kernel Ridge Regression: Matlab code and Python code
    • Section 4.4.4 Summary of Section 4.4
  • Section 4.5 Concluding Remarks
  • Section 4.6 Practical Course Material
    • Complexity-performance trade-off in spectral clustering with sparse kernel: Matlab code and Python code
    • Towards transfer learning with kernel regression: Matlab code and Python code

• Chapter 5 Large Neural Networks

  • Section 5.1 Random Neural Networks
    • Section 5.1.1 Regression with Random Neural Networks: Matlab code and Python code
    • Section 5.1.2 Delving Deeper into Limiting Kernels: Matlab code and Python code
  • Section 5.2 Gradient Descent Dynamics in Learning Linear Neural Nets: Matlab code and Python code
  • Section 5.3 Recurrent Neural Nets: Echo-State Networks: Matlab code and Python code
  • Section 5.4 Concluding Remarks
  • Section 5.5 Practical Course Material: performance of large-dimensional random Fourier features Matlab code and Python code

• Chapter 6 Large-Dimensional Convex Optimization

  • Section 6.1 Generalized Linear Classifier: Matlab code and Python code
  • Section 6.2 Large-Dimensional Support Vector Machines
  • Section 6.3 Concluding Remarks
  • Section 6.4 Practical Course Material: phase retrieval Matlab code and Python code

• Chapter 7 Community Detection on Graphs

  • Section 7.1 Community Detection in Dense Graphs
    • Section 7.1.1 The Stochastic Block Model: Matlab code and Python code
    • Section 7.1.2 The Degree-Correlated Stochastic Block Model: Matlab code and Python code
  • Section 7.2 From Dense to Sparse Graphs: A Different Approach: Matlab code and Python code
  • Section 7.3 Concluding Remarks
  • Section 7.4 Practical Course Material: Asymptotic Gaussian fluctuations of the SBM dominant eigenvector

• Chapter 8 Universality and Real Data: Matlab code and Python code

  • Section 8.1 From Gaussian Mixtures to Concentrated Random Vectors and GAN Data
  • Section 8.2 Wide-Sense Universality in Large-Dimensional Machine Learning
  • Section 8.3 Discussions and Conclusions