Generative vs. discriminative models, Maximum likelihood estimation for density estimation, Autoregressive models and sequential generation, Latent variable models, Evaluation metrics (log-likelihood, FID, IS scores),...
Generative vs. discriminative models, Maximum likelihood estimation for density estimation, Autoregressive models and sequential generation, Latent variable models, Evaluation metrics (log-likelihood, FID, IS scores), Mode collapse and posterior collapse problems, Training instabilities and divergence measures.
VAE architecture (encoder-decoder, KL divergence), Reparameterization trick and -VAE variants, Hierarchical VAEs and VQ-VAE, Denoising Diffusion Probabilistic Models (DDPM), Forward/reverse diffusion process, Denoising score matching, Classifier-free guidance, Latent diffusion models (Stable Diffusion).
Decoder-only transformers (GPT series evolution), Rotary Position Embeddings (RoPE), Grouped-Query Attention (GQA), Mixture of Experts (MoE) scaling, Context window extension techniques (RoPE scaling, ALiBi), FlashAttention and memory-efficient attention, Speculative decoding and key-value caching optimizations.
Self-supervised pretraining objectives (causal LM, masked LM, prefix LM), Chinchilla scaling laws and compute-optimal training, Instruction tuning and dataset curation, Reinforcement Learning from Human Feedback (RLHF), Proximal Policy Optimization (PPO) for alignment, Direct Preference Optimization (DPO), Constitutional AI and self-improvement.
Retrieval-Augmented Generation (RAG) architectures, Agentic workflows (ReAct, Toolformer), Multimodal generation (CLIP, DALL-E, Flamingo), Text-to-image/video/speech generation, Guardrails and content moderation, Model evaluation (human preference, Elo rankings), Production deployment (vLLM, TGI, quantization, distillation), Ethical considerations and safety measures.