MSDS-631

Deep learning is a sub-field of machine learning that focuses on learning complex, hierarchical feature representations from raw data. Over the past decade, deep learning has been remarkably successful at solving a massive set of problems on data types including images and sequential data. This success drove the extension of deep learning to other discrete domains such as sets, point clouds , graphs, and 3D shapes.

This course introduces the student to a range of topics and concepts in deep learning including the foundation neural networks, most common neural network architectures such as MLP, convolutional neural networks, and recurrent neural networks to name a few. We will also go over advanced topics such as generative models, geometric deep learning and graph neural networks. The course covers a practical aspects of deep learning and students get to learn how to use pytroch for creation/training/ineference of various networks. Intuition, mathematical notions as well as the practical aspects are all emphasized throughout the course and by the end of the course the student should have a solid theoerical and practical foundation of deep learning

Class details

INSTRUCTOR. Mustafa Hajij. San Francisco-101 Howard 155

INSTRUCTION FORMAT. Class runs for 1:50 hours 2 day/week. Instructor-student interaction during lecture is encouraged and we'll mix in mini-exercises, final project and attendance. All programming will be done in the Python 3 programming language, unless otherwise specified.

TARDINESS. Please be on time for class. It is a big distraction if you come in late.

Grade distribution :

• Final Project 50 % (see below for details)
• Homeworks 40 %
• Attendance and particupation 6 %.
• Quiz 4 %

Syllabus

Note material Additional does not optional. These are material that will not be covered in class but might probably be needed to finish your homwork.

• Day 1:

  • Review1 Linear Algebra for DL (notebook)
  • Review2 Probability for DL (notebook)
  • Review3 Some basic math concepts
  • Review4 MLE
  • Lecture 1 Slide An introduction to NNs
  • Fundamental algorithm The gradient descent (notebook)
  • Pytorch Introduction to Pytorch (notebook)
  • A visual introduction to NN playground
  • How does a DL model transform the input data to the final prediction ? Youtube video
  • What is a neural network ? YouTube
  • A minimal implementation of a NN notebook
  • Train your first NN MLP and dense layers (notebook)
  • Additional : The gradient descent algorithm for dense layers GD (notebook)
  • Additional : Stochastic Gradient Decent Notebook
  • Interview Questions notebook

• Day 2:

  • Lecture 2 Slide Neural Network as a classifier, backprop and universaltity of NN
  • An illustration of the sigmoid function notebook
  • Why function composition is useful in DL notebook
  • Functions describe the world : YouTube
  • Why neural networks can learn anything? Youtube
  • The universal approximation theorem for neural network visual
  • A video illustration of the universal approximation theorem of NNs Youtube
  • A minimal introduction to automatic differentiation Notebook

• Day 3:

  • Lecture 3 Slide Conv networks
  • Introduction to conv and pooling layers using Numpy Notebook
  • An introduction to CNNs MNIST example (notebook)
  • Visualization toolbox for convnets URL1 URL2
  • Saliency map
  • Resnets and inception modules Notebook

• Day 4:

  • MLE review MLE
  • The maximal likelihood estimation MLE and its relations to other loss functions (notebook)
  • MLE and Cross Entropy
  • Cross entropy and KL divergence
  • MLE cross entropy and KL divergence
  • Lecture Slide 4 : Probabilistic Deep Learning
  • Mixture density networks PDL application in finance

• Day 5:

  • Lecture 5 Word2Vec PDF
  • Word2Vec Tutorial
  • Lecture 5 Introduction to Language Models PDF
  • Word2Vec notebook
  • Gensim Word2Vec notebook
  • Very Simple Language Model notebook
  • Minimal RNN notebook
  • Interview Questions file

• Day 6:

  • nn.Embedding notebook
  • nn.Embedding from scratch notebook
  • Transformers notebook
  • Bert notebook
  • Lecture 6 Transformers PDF
  • Transformers Tutorials
  • Vision Transfomer pdf ViT github
  • 3blue1brown pn the transformers video
  • A great blog article about transformers article
  • Lilian Weng blog article about attention
  • Positional Encoder

• Day 7:

  • Review Cross Entropy in Pytorch notebook
  • Let's build the GPT Tokenizer by Andrej Karpathy
  • Lecture 7 Introduction to LLM PDF
  • HOW RLHF works article
  • GPT notebook
  • Lets reproduce GPT2

• Day 8:

  • Random Variable Generation (and how to generate samples from known distributions) notebook
  • Quiz questions file

• Day 9:

  • AutoEncoders notebook
  • Variational AutoEncoders notebook
  • Lecture 9 AE and VAE lecture note

• Day 10

  • NF blog
  • NF blog2
  • NF notebook
  • NF Lecture PDF
  • GANS notebook
  • GANS Lecture PDF
  • GANS blog

• Day 11

  • Graph Neural Networks (GNNs) Lecture PDF
  • Minimal GCN notebook
  • Minimal graph attention network notebook
  • GNN Tutorial notebook

• Day 12

  • Lecture 12 PDF
  • DDQN notebook
  • DQL notebook
  • DQL paper

Additional

• Common loss functions in Pytorch notebook

Quiz (Total 4 %)

In class quiz. Review questions for the quiz

Attendance and particupation (Total 6 %)

HWs (Total 40 %)

• HW1 (Due 13th of April)
• HW2 (Due 20th of April)
• HW3, you may use this file to help in the HW (Due 27th of April)
• HW4 (Due 8th of May)

Final Project (Total 50%)

Task 1: The Project Proposal, Data Selection, and Data Description (Total 5%) (Due 13th of April)

The Project Proposal, Data Selection, and Description component hold a weightage of 5% in determining your final course grade. This stage of the project requires you to submit a well-structured project proposal that encompasses several key elements:

Project Proposal:

You need to outline the objectives and goals of your project clearly. Explain the problem you intend to address using deep learning techniques and describe the overall approach you plan to take.

Data Selection:

Selecting appropriate data is crucial for the success of your project. You should detail the sources from which you will obtain your data and explain why these sources are suitable for your project. Additionally, discuss any preprocessing steps that may be required.

Data Description:

Provide a comprehensive description of the data you will be working with. This includes information about the data format, size, attributes, and any inherent challenges or limitations associated with the data. Clearly state how the selected data aligns with your project objectives.

Task 2: Jupyter notebook and Medium Post (Total 35%) (Due 14th of May)

This project requires you to create a comprehensive and well-structured Jupyter notebook that effectively presents your work. The notebook should include the following components:

Data Preprocessing (5%)

Describe the methods and steps employed to preprocess and prepare the data for your deep learning model. This may involve tasks such as data cleaning, feature engineering, data augmentation, or any other relevant preprocessing techniques.

Model Implementation (10%)

Detail the architecture and implementation details of your deep learning model. Include code snippets, well-commented code cells, or references to external code repositories if applicable. Explain the rationale behind your model choices and any modifications or enhancements you made to existing models.

Methods (5%)

Provide a clear description of the methodologies used in your project. Explain the algorithms, techniques, or frameworks employed, ensuring that your approach is well-documented and reproducible.

Experiments and Results (10%)

Present the experiments conducted during your project and report the obtained results. Include relevant performance metrics, accuracy scores, loss curves, or any other measurements used to evaluate your model's performance. Use tables, graphs, or visualizations to effectively communicate your experimental findings.

Medium Post (10%)

Write a Medium post about your project, covering the same topics as the project proposal but in a more narrative and engaging format. The Medium post should be structured to attract readers and provide them with a clear understanding of your project's objectives, data selection process, and the significance of your work.

Please ensure that your Jupyter notebook is well-documented and organized, making it easy for others to understand and reproduce your work. The total grade for the project is out of 30% of your final course grade.

Task 3: Recorded presentation (10%) (Due 14th of May)

The recorded presentation contributes 10% to your overall course grade. You will be required to deliver a presentation, recorded in video format, where you showcase and explain your project, including its objectives, methodology, results, and conclusions. This should not exceed 10 minutes and should be presented by all team members.​

Deep Learning Project Ideas

Here are 20 project ideas for deep learning:

1. Image Classification: Build a deep learning model to classify images into various categories.
2. Object Detection: Create a model to detect objects in images or videos and draw bounding boxes around them.
3. Sentiment Analysis: Develop a model to analyze the sentiment (positive, negative, neutral) of text data such as reviews or social media posts.
4. Language Translation: Build a deep learning model to translate text from one language to another.
5. Facial Recognition: Create a model that can recognize and identify faces in images or videos.
6. Recommendation System: Develop a recommendation system using deep learning to suggest personalized recommendations for users.
7. Text Generation: Train a deep learning model to generate text, such as song lyrics, poems, or story paragraphs.
8. Anomaly Detection: Build a model that can detect anomalies or outliers in datasets.
9. Speech Recognition: Create a model that can transcribe spoken words into written text.
10. Fraud Detection: Develop a deep learning model to detect fraudulent transactions or activities.
11. Image Segmentation: Build a model to segment images into different regions or objects.
12. Emotion Recognition: Create a model that can recognize and classify emotions from facial expressions.
13. Time Series Prediction: Develop a model that can predict future values in time series data, such as stock prices or weather patterns.
14. Medical Diagnosis: Build a model for diagnosing medical conditions or diseases based on medical images or patient data.
15. Style Transfer: Create a model that can transfer the style of one image to another while preserving content.
16. Speech Synthesis: Train a deep learning model to generate human-like speech.
17. Video Action Recognition: Develop a model to recognize and classify actions or activities in videos.
18. Music Generation: Train a deep learning model to compose original music pieces.
19. Image Super-Resolution: Build a model to enhance the resolution and quality of low-resolution images.
20. Self-Driving Cars: Develop a deep learning model to control a simulated or real autonomous vehicle.

These project ideas cover a wide range of modern and interesting applications that can be implemented using deep learning techniques. Choose the one that intrigues you the most and embark on your deep learning project!