AI has enabled remarkable progress in diagnosis across many clinical domains. However, existing benchmarks and models are primarily limited to medical text and imaging, which hinders the development of large-scale multimodal methods that can make holistic assessments of patient health and well-being. Our research advances the ability for AI models to unify diverse clinical data across text, imaging, video, medical and physiological sensing, longitudinal observations, and graphical networks. These technologies can assist medical professionals in accurate and efficient decision-making, while helping patients track and improve their overall physical, emotional, and social wellbeing from rich multimodal indicators.
Among our contributions are CLIMB (Clinical Large-scale Integrative Multi-modal Benchmark), one of the most comprehensive collections of clinical data ever compiled for AI research. CLIMB unifies 4.51 million patient samples spanning medical text, imaging (X-rays, CT scans), medical sensing (ECG heart rhythms), brain wave patterns, and molecular structures.
Using CLIMB, we trained QoQ-Med, the first open generalist clinical foundation model that jointly reasons across medical images, time-series signals, and text reports using a new reasoning objective that better handles modality and task heterogeneity. A promising benefit of these reasoning models is their ability to explain their predictions over multiple steps, highlight salient parts of the medical image and time-series to support their conclusions, and perform differential diagnosis compared to other possible conditions that share similar symptoms.
We are actively collaborating with the clinical researchers at Harvard Medical School and Johns Hopkins University School of Medicine to evaluate and deploy these multimodal clinical foundation models in practice, and developing models that can remain robust across varying patient demographics and regions.
Key works:
CLIMB: Data Foundations for Large-Scale Multimodal Clinical Foundation Models, ICML 2025
FairGRPO: Towards Fair Reasoning Foundation Models for Clinical Diagnosis, arXiv 2025