Information Geometry, Statistics and Learning

This repository contains coursework and project materials for the Information Theory, Statistics and Learning course.

Repository Structure

• final/: Final exam Solutions and materials
• Problem set 1/: Solutions and materials for Problem Set 1
• Problem set 2/: Solutions and materials for Problem Set 2
• Problem set 3/: Solutions and materials for Problem Set 3
• Problem set 4/: Solutions and materials for Problem Set 4
• Project/: Course project materials and latex report
• Course project report: Report for the final course project

Course Project

The course project explores key concepts in Information Geometry and their applications. Main topics include:

1. Introduction
2. Prerequisites
   • Manifolds
   • Vectors and covectors
   • Riemannian geometry
   • Dual geometric structures
3. Divergences
   • Fisher metric
   • Divergence from a manifold connection
   • Bregman divergence
   • Generalized Pythagoras theorem
4. Projections
   • Projection definition
   • e-projection and m-projection
   • MLE and m-projection
   • Maximum entropy principle
5. Applications
   • Overcomplete analysis
   • Natural Gradient Descent (NGD)
   • NGD implementation
   • Posterior distribution approximation

For more details, please refer to the project report in the Project/ directory.