MMAI4Humans | Schedule

** Exact topics and schedule subject to change, based on interests and time. **

Part	Topics	Readings
1	Introduction [slides] What is Multimodal? Definitions, dimensions of heterogeneity and cross-modal interactions. Historical view and multimodal research tasks. Core technical challenges: representation, alignment, transference, reasoning, generation, and quantification.	Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
2	Multimodal challenges Why is multimodal hard? Introduction to core challenges. Overview of multimodal representation, alignment, reasoning, transfer, generation, and quantification. Identifying recent solutions for practitioners.	Multimodal interaction: A review Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think! The Platonic Representation Hypothesis
3	Recent advances in multimodal AI Multimodal transformers and foundation models Multimodal generative models Multimodal agents	LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention Scaling Laws for Generative Mixed-Modal Models NExT-GPT: Any-to-Any Multimodal LLM
4	Multimodal AI for Human Sensing [slides] Sensor data synthesis: Video to Doppler, Video to IMU, Video to Audio, MoCap to IMU, MoCap to UWB Data augmentation Temporal data modeling	Synthesizing Doppler Radar data for Privacy-Aware Activity Recognition MobilePoser: Real-Time Full-Body Pose Estimation and 3D Human Translation from IMUs in Mobile Consumer Devices
5	Ethics, interpretability and privacy Privacy and fairness concerns Handling errors and uncertainty Bringing humans into the loop	SAMoSA: Sensing Activities with Motion and Subsampled Audio
6	Applications Human activity recognition, pose estimation, gesture recognition Infrastructure and environmental sensing Wellness and fitness tracking, mobile health monitoring	Direction-of-Voice (DoV) Estimation for Intuitive Speech Interaction with Smart Devices Ecosystems EduSense: Practical Classroom Sensing at Scale
7	Hardware and Sensors for Multimodal AI [slides] Challenges and opportunities in hardware and sensors for multimodal AI Importance of scalable, customizable hardware platforms Key applications benefiting from advancements in multimodal sensing and feedback interfaces	Learning the signatures of the human grasp using a scalable tactile glove
8	Multimodal sensing and feedback interface Advanced fabrication techniques for multimodal sensing hardware Addressing scalability, adaptability, and customization in hardware development Innovations in state-of-the-art data acquisition systems for multimodal interfaces Integration of diverse sensing modalities into compact, flexible form factors	Adaptive Tactile Interaction Transfer via Digitally Embroidered Smart Gloves Skin-integrated wireless haptic interfaces for virtual and augmented reality
9	Multisensory data fusion Approaches to synchronize and interpret multimodal data Strategies for enhancing signal quality and accuracy Future directions enabled by multisensory interfaces	ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment 3D-ViTac Learning Fine-Grained Manipulation with Visuo-Tactile Sensing