Deep Learning Topics and Resources

Resources for DL in General

1. Blogs
   • Lilian Weng’s Blog [link]
   • AI Summer Blog [link]
   • Colah’s Blog [link]
2. Books
   • Neural Networks and Deep Learning [link]
   • Deep Learning Book [link]
   • Dive into Deep Learning [link]
   • Reinforcement Learning: An Introduction | Sutton and Barto [link]
3. Open Courses
   • CS-229 Machine Learning Stanford | Andrew Ng [youtube] [website]
   • CS-231n Computer Vision Stanford [youtube] [website]
   • CS-224n Natural Language Processing [youtube] [website]
   • Introduction to Reinforcement Learning with David Silver [youtube] [website]

Mathematics

1. Linear Algebra ([notes][practice questions])

   • 3Blue1Brown essence of linear algebra [youtube]
   • Gilbert Strang’s lectures on Linear Algebra [link] [youtube]
   • Topics
     • Linear Transformations
     • Linear Dependence and Span
     • Eigendecomposition - Eigenvalues and Eigenvectors
     • Singular Value Decomposition [blog]

2. Probability and Statistics ([notes][youtube series])

   • Harvard Statistics 110: Probability [link] [youtube]
   • Topics
     • Expectation, Variance, and Co-variance
     • Distributions
     • Random Walks
     • Bias and Variance
       • Bias Variance Trade-off
     • Estimators
       • Biased and Unbiased
     • Maximum Likelihood Estimation [blog]
     • Maximum A-Posteriori (MAP) Estimation [blog]

3. Information Theory [youtube]

   • (Shannon) Entropy [blog]
   • Cross Entropy, KL Divergence [blog]
   • KL Divergence
     • Not a distance metric (unsymmetric)
     • Derivation from likelihood ratio (Blog)
     • Always greater than 0
       • Proof by Jensen's inequality (Stack Overflow Link)
     • Relation with Entropy (Explanation)

Basics

1. Neural Networks Overview [youtube]
2. Backpropogation
   • Vanilla [blog]
   • Backpropagation in CNNs [blog]
   • Backprop through time [blog]
3. Loss Functions
   • MSE Loss
     • Derivation by MLE and MAP
   • Cross Entropy Loss
     • Binary Cross Entropy
     • Categorical Cross Entropy
4. Activation Functions (Sigmoid, Tanh, ReLU and variants) (blog)
5. Optimizers
6. Regularization
   • Early Stopping
   • Noise Injection
   • Dataset Augmentation
   • Ensembling
   • Parameter Norm Penalties
     • L1 (sparsity)
     • L2 (smaller parameter values)
   • BatchNorm [Paper]
     • Internal Covariate Shift
     • BatchNorm in CNNs [Link]
     • Backprop through BatchNorm Layer [Explanation]
   • Dropout Regularization [Paper]

Computer Vision

1. Convolution [youtube]

   • Cross-correlation
   • Pooling (Average, Max Pool)
   • Strides and Padding
   • Output volume dimension calculation
   • Deconvolution (Transposed Convolution), Upsampling, Reverse Pooling [Visualization]
   • Types of convolution operation [blog]

2. ImageNet Classification

   • AlexNet [paper] [blog]
   • ZFNet [paper] [blog]
   • VGGNet [paper] [blog]
   • InceptionNet [paper] [blog]
   • ResNet [paper] [blog]
   • DenseNet [paper] [blog]
   • SENet [paper] [blog]
   • ViT [paper] [blog]
   • Swin Transformer [paper] [blog]
   • BEiT [paper] [blog]
   • ConvNext [paper] [blog]

3. Object Detection [blog series]

   • RCNN [paper]
   • Fast RCNN [paper]
   • Faster RCNN [paper]
   • Mask RCNN [paper]
   • YOLO (Real-time object recognition) [blog]
   • SSD (Single Shot Detection) [paper]
   • DETR [project page] [annotated DETR]

4. Semantic Segmentation

   • UNet [paper]
   • DeepLab [paper]
   • MaskFormer [paper] [project page]

Natural Language Processing

1. Recurrent Neural Networks

   • Architectures (Limitations and inspiration behind every model)
     • Vanilla [blog]
     • GRU, LSTMs [blog_1] [blog_2]
     • Bidirectional
   • Vanishing and Exploding Gradients

2. Word Embeddings [blog_1] [blog_2]

   • Word2Vec
   • CBOW
   • Glove
   • SkipGram, NGram
   • FastText
   • ELMO
   • BERT

3. Transformers [blog posts] [youtube series]

   • Attention is All You Need [blog] [paper] [annotated transformer]
   • Query-Key-Value Attention Mechanism (Quadratic Time)
   • Position Embeddings [blog]
   • BERT (Masked Language Modelling) [blog]
   • Longe Range Sequence Modelling [blog]
   • ELECTRA (Pretraining Transformers as Discriminators) [blog]
   • GPT (Causal Language Modelling) [blog]
   • OpenAI ChatGPT [blog]

Multimodal Learning

• Vision Language Models | AI Summer [blog]
• Open AI DALL-E [blog]
• OpenAI CLIP [blog]
• Flamingo [blog]
• Gato [blog]
• data2vec [blog]
• OpenAI Whisper [blog]

Generative Models

1. Generative Adversarial Networks (GANs) [blog series]
   • Basic Idea
   • Variants
     • Vanilla GAN [paper]
     • DCGAN [paper]
     • Wasserstein GAN [paper]
     • Conditional GAN [paper]
   • Mode Collapse
   • GAN Hacks [link]
2. Variational Autoencoders (VAEs)
   • Variational Inference [tutorial paper]
   • ELBO and Loss Function derivation
3. Normalizing Flows
   • Basic Idea and Applications [link]

Stable Diffusion

• Demos

  • Lexica (Stable Diffusion search engine) [link]
  • Stability AI | Huggingface Spaces [link]

• Diffusion Models in general [paper]

  • What are Diffusion Models? | Lil'Log [link]

• Stable Diffusion | Stability AI [blog] [annotated stable diffusion]

• Illustrated Stable DIffusion | Jay Alammar [blog]

• Stable Diffusion in downstream Vision tasks

  • DiffusionDet [paper]

Keeping up with the developments in Deep Learning

• Youtube Channels
  • Yannic Kilcher [link]
  • Two Minute Papers [link]
• Blogs
  • DeepMind Blog [link]
  • OpenAI Blog [link]
  • Google AI Blog [link]
  • Meta AI Blog [link]
  • Nvidia - Deep Learning Blog [link]
  • Microsoft Research Blog [link]
• Trending Reseach Papers
  • labml [link]
  • deep learning monitor [link]

Contributing

We welcome contributions to add resources such as notes, blogs, or papers for a topic. Feel free to open a pull request for the same!