Module 02: Deep Learning

Goal

Understand neural networks well enough to train, debug, regularize, evaluate, export, and serve a model.

Core Topics

• Perceptron and multilayer neural networks.
• Forward propagation and backpropagation.
• Activation functions: ReLU, sigmoid, tanh, GELU.
• Loss functions and optimizers.
• Batch size, epochs, learning rate, schedulers.
• Regularization, dropout, batch normalization, weight decay.
• CNNs for images.
• RNNs and LSTMs for sequences.
• Attention as a bridge to transformers.

Study Tasks

• Build a neural network from scratch for a toy dataset.
• Train a classifier with PyTorch or TensorFlow.
• Track experiments and compare hyperparameters.
• Export a trained model.
• Build a minimal inference API.
• Containerize the API.

Debugging Drills

• Train with a learning rate that is too high and observe divergence.
• Train with a learning rate that is too low and observe slow convergence.
• Remove regularization and observe overfitting.
• Use a tiny dataset to verify the model can overfit intentionally.

Project

Build projects/02-neural-network-classifier/.

Tips

• Start with a tiny model and prove the pipeline works.
• Check data shapes constantly.
• Plot loss curves before changing model architecture.
• Do not tune many hyperparameters at once.

Recommended Resources

• PyTorch tutorials.
• TensorFlow/Keras guides.
• Deep Learning by Goodfellow, Bengio, and Courville for reference.
• fast.ai for practical intuition.

Completion Checklist

• Can explain forward propagation.
• Can explain backpropagation at a high level.
• Can diagnose vanishing gradients and overfitting.
• Can train and serve a small model.
• Can document experiments and results.