Lecture Slides for Deeplearning book

This repo contains lecture slides for Deeplearning book. This project is maintained by InfoLab @ DGIST (Large-scale Deep Learning Team), and have been made for InfoSeminar. It is freely available only if the source is marked.

The slides contain additional materials which have not detailed in the book.
Also, some materials in the book have been omitted.

Maintained by

InfoLab @ DGIST(Daegu Gyeongbuk Institute of Science & Technology)

Coverage

This repo covers Chapter 5 to 20 in the book.

Credits

Name	Chapters
Jinwook Kim	5-1, 8, 16, 17
Heechul Lim	9-2, 11, 12, 15, 19
Hyun-Lim Yang	5-2, 7-2, 10-2
Keonwoo Noh	7-1, 9-1, 14, 20
Eunjeong Yi	6, 10-1, 13, 18

The contents of lecture slides

Part 1. Applied Math and Machine Learning Basics

5-1. Machine Learning Basics

• Learning algorithms
• Capacity, overfitting and underfitting
• Hyperparameters and validation sets
• Estimators, bias and variance
• Maximum likelihood estimation

5-2. Machine Learning Basics

• Bayesian statistics
• Supervised learning algorithms
• Unsupervised learning algorithms
• Stochastic gradient descent

Part 2. Deep Networks: Modern Practices

6. Deep Feedforward Networks

• Example: Learning XOR
• Gradient-Based Learning
• Hidden Units
• Architecture Design
• Back-Propagation and Other Differentiation

7-1 Regularization for Deep Learning

• Parameter Norm Penalties
• Norm Penalties as Constrained Optimization
• Regularization and Under-Constrained Problems
• Dataset Augmentation
• Noise Robustness
• Semi-Supervised Learning
• Multitask Learning
• Early stopping

7-2 Regularization for Deep Learning

• Parameter Tying and Parameter Sharing
• Bagging and Other Ensemble Methods
• Dropout
• Adversarial Training

8 Optimization for Training Deep Models

• How Learning Differs from Pure Optimization
• Challenges in Neural Networks
• Basic Algorithms
• Algorithms with Adaptive Learning Rates
• Parameter Initialization Strategies
• Approximate Second-order Methods
• Optimization Strategies and Meta-algorithms

9-1 Convolutional Networks

• The Convolution Operation
• Motivation
• Pooling
• Convolution and Pooling as an Infinitely Strong Prior
• Variants of the Basic Convolution Function

9-2 Convolutional Networks

• Structured Outputs
• Data Types
• Efficient Convolution Algorithms
• Random or Unsupervised Features
• The Neuroscientific Basis for Convolutional Networks

10-1 Sequence modeling: Recurrent and Recursive Nets

• Unfolding Computational Graphs
• Recurrent Neural Networks
• Bidirectional RNNs
• Encoder-Decoder Sequence-to-Sequence Architectures
• Deep Recurrent Networks
• Recursive Neural Networks

10-2 Sequence modeling: Recurrent and Recursive Nets

• The challenge of Long-term
• Echo State Networks
• Leaky Units and Other strategies for Multiple Time Scales
• The Long Short-Term Memory and Other Gated RNNs
• Optimization for Long-Term Dependencies
• Explicit Memory

11, 12 Practical Methodology and Applications

• Performance Metrics
• Default Baseline Models
• Determining Whether to Gather More Data
• Selecting Hyperparameters
• Debugging Strategies
• Computer Vision

Part 3. Deep Learning Research

13 Linear Factor Models

• Probabilistic PCA and Factor Analysis
• Independent Component Analysis
• Sparse Coding

14 Autoencoders

• Introduction
• Stochastic Encoders and Decoders
• Regularized autoencoders
• Representational Power, Layer Size and Depth

15 Representation Learning

• Unsupervised pre-training
• Introduction of supervised(SL) and unsupervised learning(UL)
• Representation
• Clustering
• K-means
• Gaussian Mixture Model
• EM algorithm
• Practical example

16, 17 Structured Probabilistic Models for Deep Learning and Monte Carlo Methods

• The Challenge of Unstructured Modeling
• Using Graphs to Describe Model Structure
• Sampling from Graphical Models
• The Deep Learning Approach to Structured Probabilistic Models
• Sampling and Monte Carlo Methods
• Markov Chain Monte Carlo Methods
• Gibbs Sampling

18 Confronting the Partition Function

• The Log-Likelihood Gradient
• Stochastic Maximum Likelihood and Contrastive Divergence
• Estimating the Partition Function

19 Approximate Inference

• Approximation
• Maximum Likelihood(MLE) and Maximum A Posteriori(MAP)
• Inference
• Taxonomy of deep generative models
• KL-Divergence
• Variational Inference

20 Deep Generative Models

• Generative models
• Boltzmann Machines
• Restricted Boltzmann Machines
• Deep Belief Networks

---

License: CC-BY