** Exact topics and schedule subject to change, based on student interests and course discussions. **

Date Topics Readings
2/3 Week 1 Course introduction
  • Multimodal core challenges
  • Course syllabus
2/5 Week 1 Multimodal datasets
  • Research tasks and datasets
  • Intro to AI research
2/10 Week 2 Datasets tutorial
  • Data processing and visualization
  • Pytorch and modeling
2/12 Week 2 Unimodal representations
  • Dimensions of heterogeneity
  • Common model architectures
2/17 Week 3 Multimodal fusion
  • Cross-modal interactions
  • Early and late fusion
2/19 Week 3 Explainable fusion
  • Dynamic and explainable fusion
  • Attention-based fusion
2/24 Week 4 Fusion tutorial
  • Higher-order interactions
  • Multimodal fusion models
2/26 Week 4 Multimodal transformers
  • Self-attention & transformers
  • Multimodal transformers
3/3 Week 5 Multimodal foundation models
  • Multimodal pre-training
  • Multimodal fine-tuning
3/5 Week 5 Multimodal LLMs tutorial
  • Instruction tuning
  • Fine-tuning approaches
3/10 Week 6 Multimodal alignment
  • Multimodal grounding
  • Aligned representations
3/12 Week 6 Cross-modal transfer
  • Modality transfer and co-learning
  • Self-training and multitask learning
3/17 Week 7 Multimodal generation
  • Translation, summarization, creation
  • Model evaluation and ethics
3/19 Week 7 In-class midterm
3/24 Week 8 Spring Break – No lectures
3/26 Week 8 Spring Break – No lectures
3/31 Week 9 Multimodal reasoning
  • Reinforcement learning
  • Multi-step reasoning
4/2 Week 9 Reasoning tutorial
  • Reasoning reward design
  • Reinforcement learning training
4/7 Week 10 Prescriptive modeling
  • Optimization & prediction
  • Causal and counterfactual
4/9 Week 10 Multimodal interaction
  • Interactive agents
  • Interactive reasoning
4/14 Week 11 Agents tutorial
  • Multimodal agent pipelines
  • Agent evaluation
4/16 Week 11 Multimodal & design
  • Multimodal agents
  • Applications in design
4/21 Week 12 Human-AI interaction
  • Interaction mediums
  • Human-in-the-loop and safety
4/23 Week 12 Multimodal & manufacturing
  • Multisensor fusion
  • Applications in manufacturing
4/28 Week 13 Multimodal & cities
  • Spatial and temporal fusion
  • Applications in cities
4/30 Week 13 Multimodal & transportation
  • Multimodal agents
  • Applications in transportation
5/5 Week 14 New research directions
  • Recent research in multimodal AI
5/7 Week 14 New research directions
  • Recent research in multimodal AI
5/12 Week 15 Project presentations