Track 03: Landmark Models — BERT, GPT & Mixture-of-Experts

Track 02 built the transformer block in the abstract. This track shows the two directions the field split it into — BERT's masked, bidirectional encoder for understanding text, and GPT's causal, decoder-only generation for producing it — why the generative path won for the chat-and-agents era, and how Mixture-of-Experts architectures scale parameter count without scaling compute proportionally.

Modules

  1. The Pretraining Paradigm
  2. BERT: Masked Language Modeling
  3. BERT Architecture Deep Dive
  4. Fine-Tuning BERT for Classification
  5. BERT Variants: RoBERTa, DistilBERT, DeBERTa
  6. Sentence Transformers and Bi-Encoders
  7. GPT: Causal Language Modeling
  8. GPT-1 and GPT-2
  9. GPT-3 and In-Context Learning
  10. GPT-4 and Beyond
  11. Why Decoder-Only Won
  12. Llama and the Open-Weight Lineage
  13. Mixture of Experts: The Core Idea
  14. MoE Routing and Load Balancing
  15. Mixtral and MoE in Practice
  16. How to Read a Model Card
  17. Capstone Project