Track 16: Speech & Multimodal Models

Beyond text: how audio becomes something a model can process, speech recognition (Whisper), text-to-speech, the full realtime voice-agent pipeline and its latency budget, and multimodal LLMs that take images and video as first-class input alongside text.

Modules

1. Audio as Data: Waveforms and Spectrograms
2. Speech Recognition Fundamentals
3. Whisper Architecture
4. Running Whisper in Practice
5. Streaming and Realtime Transcription
6. Speaker Diarization
7. Text-to-Speech Fundamentals
8. Neural TTS and Voice Cloning
9. TTS APIs in Practice
10. The Voice Agent Pipeline (STT → LLM → TTS)
11. Latency Budgets for Voice
12. Realtime Speech-to-Speech APIs
13. Vision: How LLMs See Images
14. Vision Use Cases: OCR, Documents, UI
15. Video Understanding
16. Capstone Project