MIT MMAI | Schedule

** Exact topics and schedule subject to change, based on student interests and course discussions. **

Date	Topics	Readings
2/3	Week 1.1 Course introduction [slides] [video] Multimodal core challenges Course syllabus	Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions Multimodal Machine Learning: A Survey and Taxonomy Representation Learning: A Review and New Perspectives
2/5	Week 1.2 Multimodal datasets [slides] [video] Research tasks and datasets Intro to AI research	Machine learning: Trends, Perspectives, and Prospects Representation Learning: A Review and New Perspectives Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
2/10	Week 2.1 AI tutorial [slides] [video] Data processing and visualization Pytorch and AI modeling	A Recipe for Training Neural Networks Fine-tuning a Code LLM on Custom Code on a single GPU MAS.S60 Pytorch Introduction
2/12	Week 2.2 Unimodal representations [slides] [video] Dimensions of heterogeneity Common model architectures	Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Attention Is All You Need Neural Machine Translation by Jointly Learning to Align and Translate Deep Sets Graph Attention Networks
2/17	Week 3.1 No class, shifted President's day
2/19	Week 3.2 Multimodal fusion [slides] [video] Early and late fusion Explainable fusion	Ten Myths of Multimodal Interaction Multimodal interaction: A review
2/24	Week 4.1 More fusion [slides] [video] Higher-order interactions Multimodal fusion models	Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think! Kosmos-2: Grounding Multimodal Large Language Models to the World Chameleon: Mixed-modal early-fusion foundation models MM1: Methods, Analysis and Insights from Multimodal LLM Pre-training MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
2/26	Week 4.2 Multimodal alignment [slides] [video] Multimodal grounding Aligned representations	What Makes for Good Views for Contrastive Learning? Characterization and classification of semantic image-text relations When and why vision-language models behave like bags-of-words, and what to do about it? The Platonic Representation Hypothesis Understanding the Emergence of Multimodal Representation Alignment Does equivariance matter at scale? Learning Transferable Visual Models From Natural Language Supervision? Emerging Properties in Self-Supervised Vision Transformers
3/3	Week 5.1 Large multimodal models [slides] [video] Multimodal transformers Pre-training & fine-tuning	Training Compute-Optimal Large Language Models Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks LoRA: Low-Rank Adaptation of Large Language Models A Visual Guide to Mixture of Experts (MoE) A Visual Guide to Quantization Improved Baselines with Visual Instruction Tuning Gated Linear Attention Transformers with Hardware-Efficient Training Unintended Impacts of LLM Alignment on Global Representation A Visual Guide to Quantization Scaling Instruction-Finetuned Language Models
3/5	Week 5.2 Multimodal LLMs tutorial [slides] [video] Fine-tuning Instruction tuning	Multimodal Transformer for Unaligned Multimodal Language Sequences Masked Autoencoders Are Scalable Vision Learners Scaling Laws for Native Multimodal Models Scaling Laws for Native Multimodal Models Transfer between Modalities with MetaQueries LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference ModaVerse: Efficiently Transforming Modalities with LLMs Spider: Any-to-Many Multimodal LLM SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites NExT-GPT: Any-to-Any Multimodal LLM Learning to rebalance multi-modal optimization by adaptively masking subnetworks
3/10	Week 6.1 Multimodal generation [slides] [video] Translation, summarization, creation Model evaluation and ethics	Compositional Generative Modeling: A Single Model is Not All You Need Movie Gen: A Cast of Media Foundation Models
3/12	Week 6.2 Modern generative AI [slides] [video] VAEs, diffusion, flow models Controllable generation	Scalable Diffusion Models with Transformers Flow Matching in Latent Space Scaling Rectified Flow Transformers for High-Resolution Image Synthesis Large Language Diffusion Models Flow Matching for Generative Modeling Flow Matching Guide and Code FlowMotion: Target-Predictive Conditional Flow Matching for Jitter-Reduced Text-to-Motion Generation MusFlow: Multimodal Music Generation via Conditional Flow Matching Unraveling the Connections Between Flow Matching and Diffusion Probabilistic Models Exploring Diffusion and Flow Matching Under Generator Matching
3/17	Week 7.1 Midterm review [slides] [video] Multimodal fusion & alignment Multimodal foundation models
3/19	Week 7.2 In-class midterm
3/24	Week 8.1 Spring Break – No lectures
3/26	Week 8.2 Spring Break – No lectures
3/31	Week 9.1 Multimodal reasoning [slides] [video] Reinforcement learning Multi-step reasoning	Deep reinforcement learning from human preferences Deepseek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning Faulty reward functions in the wild Direct preference optimization: Your language model is secretly a reward model
4/2	Week 9.2 Explainable reasoning [slides] [video] Explainable AI LLMs for explainability
4/7	Week 10.1 Multimodal interaction [slides] [video] Interactive agents Interactive reasoning	VideoWebArena: Evaluating Multimodal Agents on Video Understanding Web Tasks OpenVLA: An Open-Source Vision-Language-Action Model
4/9	Week 10.2 Cross-modal transfer [slides] [video] Modality transfer and co-learning Self-training and multitask learning	LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day DreamLLM: Synergistic Multimodal Comprehension and Creation PaLM-E: An Embodied Multimodal Language Model
4/14	Week 11.1 Multimodal & manufacturing [slides] [video] Multisensor fusion Applications in manufacturing
4/16	Week 11.2 Multimodal & design [slides] [video] Multimodal agents Applications in design
4/21	Week 12.1 Prescriptive modeling [slides] [video] Optimization & prediction Causal & counterfactual
4/23	Week 12.2 Agents tutorial [slides] [video] Multimodal agent pipelines Agent evaluation
4/28	Week 13.1 Multimodal & cities [slides] [video] Spatial and temporal fusion Applications in cities
4/30	Week 13.2 Multimodal & transportation [slides] [video] Autonomous vehicles Applications in transportation
5/5	Week 14.1 Self-evolving AI [slides] [video] Continual learning Self-evolution and optimization
5/7	Week 14.2 AI for new senses [slides] [video] Touch, smell, taste Human-AI interaction	Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs Guidelines for Human-AI Interaction
5/12	Week 15.1 Project presentations