Conditional Multimodal MRI Synthesis & Brain Tumor Segmentation
Dual-Model Medical AI Pipeline for the BraTS 2020 Brain Tumor Challenge
Project Domain
Deep Learning & Medical Imaging
Execution Stage
Model Study
Timeline / Year
2026 · Two-Person Research Project
Repository & Code
GitHub Repository →Visual Documentation & Artifacts
Click any photo to enlargeEngineering with real-world constraints.
This project implements a dual-model medical deep learning system addressing two closely related clinical problems: automatically segmenting brain tumors across multiple anatomical sub-regions, and synthesizing missing or unavailable MRI scan modalities from existing data.
Brain tumor diagnosis and treatment planning typically requires all four MRI sequences — T1, T2, T1 contrast-enhanced (T1ce), and FLAIR. In practice, some modalities are often missing due to patient movement artifacts, scanner failures, or clinical time constraints. A model that can accurately reconstruct missing modalities from available ones has direct clinical utility.
We address both problems simultaneously: the segmentation model precisely delineates tumor boundaries used as conditioning masks, and the synthesis model generates clinically faithful, high-fidelity multi-sequence MRI volumes for any missing modality conditioned on those masks. Rather than generating superficial visual approximations, the diffusion model learns the true underlying tissue contrast distributions and pathological tissue characteristics across sequences. The project was developed as a two-person team and trained on Google Cloud TPU v3-8 infrastructure.
Clinical Context & Research Motivation
Brain tumor segmentation is a critical step in clinical workflow, providing oncologists with precise tumor volumes for treatment planning, radiotherapy contouring, and treatment response monitoring. Manual delineation of MRI scans is extremely time-consuming and subject to inter-rater variability. Automated segmentation is therefore an important research target.
Simultaneously, the availability of multiple MRI modalities is frequently incomplete in clinical practice. Acquiring all four sequences requires patients to remain motionless for an extended scan session, and technical failures affect individual sequences. A synthesis model that reconstructs any missing modality conditioned on available data reduces scan time, improves patient throughput, and enables downstream segmentation models to receive complete inputs even when raw data is incomplete.
The BraTS 2020 (Brain Tumor Segmentation) Challenge provides a well-annotated benchmark of multi-institutional MRI scans, establishing standardized evaluation criteria for both segmentation accuracy and synthesis quality. Our pipeline targets both tasks on this dataset.
Component Deep Dives
BraTS 2020 Multi-Modal Scans
│
┌──────────────┴──────────────┐
▼ ▼
ResNet-Encoded U-Net Conditional DDPM
(Multi-Class Segmentation) (4-Modality Synthesis)
│ │
[WT / TC / ET Masks] [T1, T2, T1ce, FLAIR]
│ │
└──────────────┬──────────────┘
▼
CPU-EMA Weight Tracking
(Solves TPU bf16 Underflow)Segmentation Architecture — ResNet-Encoded U-Net
The segmentation model uses a U-Net backbone with a Residual Encoder (ResNet) replacing the standard convolutional encoder. Residual connections in the encoder allow gradients to flow cleanly through deep network layers during backpropagation, preventing vanishing gradient problems and enabling the use of a significantly deeper feature extraction hierarchy.
The decoder uses symmetric skip connections from the encoder to recover fine-grained spatial detail lost during downsampling — a standard U-Net design decision that is particularly important for precise tumor boundary delineation at voxel resolution.
The model performs multi-class segmentation across three clinically defined sub-regions: Whole Tumor (WT) encompassing the complete lesion and surrounding edema, Tumor Core (TC) comprising the necrotic center and active tumor cells, and Enhancing Tumor (ET) identifying the actively proliferating region visible on contrast-enhanced T1 scans. These hierarchical regions correspond directly to clinical treatment zones.
Explore Full Codebase on GitHub
Includes complete source code, installation instructions, launch files, and documentation.
Quantitative Metrics
0.86 Dice
Enhancing Tumor (ET)
Mean Dice score on BraTS 2020 validation volumes
0.80 Dice
Whole Tumor (WT)
Mean Dice across complete lesion and surrounding edema
0.76 Dice
Tumor Core (TC)
Mean Dice on necrotic center and active tumor core
50 Steps
DDIM Sampling Speed
Accelerated inference with v-prediction objective on TPU v3-8
Key Capabilities
ResNet-Encoded U-Net Segmentation
Deep residual encoder with symmetric decoder skip connections performs precise multi-class tumor boundary delineation across Whole Tumor (WT), Tumor Core (TC), and Enhancing Tumor (ET) sub-regions.
Conditional DDPM — 4-Modality Synthesis
Jointly synthesizes T1, T2, T1ce, and FLAIR volumes simultaneously conditioned on categorical segmentation masks, learning tissue-specific cross-modal appearance distributions.
CPU-EMA Weight Tracking
Solved TPU v3-8 BFloat16 underflow during EMA accumulation by maintaining shadow weights in full Float32 precision on host CPU RAM, bypassing hardware precision limitations.
DDIM Sampling + v-Prediction Objective
50-step DDIM inference with the v-prediction noise parameterization produces high-fidelity anatomical MRI synthesis significantly faster than the standard 1000-step DDPM schedule.
Brain-Masked Normalization
Z-score normalization computed exclusively over foreground brain tissue voxels using binary brain masks, preventing background noise from distorting the intensity normalization statistics.
Multi-Institutional BraTS 2020 Benchmark
Trained and evaluated on the BraTS 2020 dataset — a multi-institutional brain MRI benchmark with standardized tumor annotation protocols across four clinical imaging modalities.
- [01]PyTorch
- [02]Diffusion Models (DDPM)
- [03]ResNet U-Net
- [04]TPU v3-8
- [05]DDIM Sampling
- [06]BraTS 2020
- [07]v-prediction
- [08]Medical Imaging
Other Selected Works
Autonomous Drone
A ROS2 autonomous-navigation stack for a DJI Tello: monocular ORB-SLAM3 for real-time visual SLAM with a custom depth-estimation and scale-recovery pipeline, plus occupancy mapping and exploration planning layers in active development.
Learn more →Computer Vision & Edge AIGesture-Controlled Drone
A real-time gesture-recognition pipeline controlling a DJI Tello in flight. Built on a self-collected dataset of 5,000+ hand-landmark samples, a feature-engineered classifier, and temporal smoothing for reliable control.
Learn more →Robotics & Embedded SystemsGesture-Controlled Robotic Hand
An MPU6050-driven glove that maps hand orientation to real-time servo control on a 4-DOF robotic arm — connecting embedded sensing directly to actuation.
Learn more →