Animal Detection & Tracking — Mixture of Experts
Dynamic Per-Frame Expert Routing for Robust Wildlife Video Detection
Project Domain
Computer Vision & Applied Deep Learning
Execution Stage
Open Source · Deployed Experiment
Timeline / Year
2026 · Applied Vision Architecture
Repository & Code
GitHub Repository →Visual Documentation & Artifacts
Click any photo to enlargeEngineering with real-world constraints.
Wildlife monitoring in natural habitat environments presents extreme visual challenges that a single detection model handles poorly. Clean, well-lit frames with stationary animals are visually simple; frames with fast-moving animals, motion blur, dense foliage occlusion, mist, or low light are visually complex and degraded. The optimal detection architecture for these two regime types is fundamentally different.
This project implements a Mixture-of-Experts (MoE) detection pipeline that dynamically routes each individual video frame to the most appropriate specialist model based on real-time analysis of that frame's image quality. A Precision Expert handles clean, sharp, high-frequency frames where detailed feature extraction pays off. A Kinetic Expert handles degraded frames where robustness to blur, noise, and occlusion is the priority.
Routing is performed by a lightweight frame-level quality estimator computing the Laplacian variance of each frame — a classical image sharpness metric that requires no neural inference and adds negligible latency. BotSort multi-object tracking maintains consistent animal identity labels across the sequence, preserving track continuity through the frame-to-frame routing switches.
Why a Mixture of Experts for Wildlife Detection?
A natural habitat wildlife video contains wildly heterogeneous frame quality within a single sequence. An animal standing in the open mid-afternoon light produces frames that are sharp, high-resolution, and visually rich — ideal conditions for a large, high-capacity detection model to extract fine-grained features and produce precise bounding boxes.
Seconds later, the same animal moves rapidly across the frame, introducing motion blur. Or it steps into shade, reducing contrast. Or it partially disappears behind vegetation. For these degraded frames, a large precision model often fails or produces unstable detections because its learned feature representations depend on high-frequency edge and texture detail that is simply absent in a blurred or occluded frame.
Training a single model to be simultaneously optimal in both regimes is extremely difficult — the regularization and augmentation strategies that help in one regime often hurt in the other. A MoE architecture avoids this tradeoff by training specialists independently for each regime and selecting between them at inference time.
Component Deep Dives
Video Frame Stream
│
▼
Laplacian Variance Filter
│
┌────────┴────────┐
Variance ≥ τ Variance < τ
(Clean / Sharp) (Blur / Motion / Fog)
▼ ▼
Precision Expert Kinetic Expert
(RT-DETR-X) (YOLO11m)
│ │
└────────┬────────┘
▼
BotSort Tracking Engine
(Kalman Persistent Animal IDs)
▼
Gradio Live DemonstrationPhase 01 — Dataset Curation: 53 → 52 Classes
The starting point is a multi-class wildlife detection dataset covering a broad taxonomy of animal species across diverse natural habitat environments. The dataset contains bounding box annotations with class labels.
Dataset inspection revealed that one class — Butterfly — suffered from severe class imbalance, containing an overwhelming excess of instances compared to all other animal classes. This extreme sample disparity skewed the loss function and biased detection thresholds away from the remaining 52 wildlife classes (larger-bodied mammals and birds).
To eliminate this severe class imbalance and ensure stable, unbiased training across the wildlife taxonomy, the Butterfly class was removed. All remaining class labels were continuously re-indexed from 0 to 51, establishing a clean, balanced 52-class dataset for both expert models.
Explore Full Codebase on GitHub
Includes complete source code, installation instructions, launch files, and documentation.
Phased Architecture
Curate 53-Class Dataset → 52 Classes
Build a balanced 52-class wildlife dataset by removing the Butterfly class due to severe class imbalance (extreme over-representation), then continuously re-indexing all remaining labels.
Train Precision Expert (RT-DETR-X)
Train the high-capacity RT-DETR-X model on clean, unaugmented structured images with standard augmentations only, specializing it for high-quality detection on favorable frames.
Train Kinetic Expert (RT-DETR / YOLO11m)
Train the robustness-focused expert on a chaos-augmented dataset simulating motion blur, fog, occlusion patches, noise injection, and extreme exposure variations.
Implement Frame-Level MoE Router
Design and implement the per-frame Laplacian variance sharpness router that routes each incoming video frame to the appropriate expert in real time without neural inference overhead.
Integrate BotSort Multi-Object Tracking
Integrate BotSort tracking with Kalman-filter motion prediction to assign and maintain persistent animal identity labels across the full video sequence.
Deploy Interactive Gradio Application
Package the full router + expert + tracking pipeline into an interactive Gradio application with visualized routing decisions, track ID overlays, and an adjustable threshold slider.
Key Capabilities
53-Class → 52-Class Dataset Balancing
Resolved severe class imbalance by removing the heavily over-represented Butterfly class and continuously re-indexing the remaining 52 wildlife categories for stable, unbiased training.
Precision Expert: RT-DETR-X on Clean Data
High-capacity transformer detection model trained on clean images to specialize in fine-grained species discrimination and precise bounding box regression under favorable conditions.
Kinetic Expert: Chaos-Augmented Robustness
Trained on a synthetically degraded dataset simulating motion blur, fog, occlusion, and noise to produce reliable detections under difficult field conditions.
Zero-Overhead Laplacian Variance Router
Classical image sharpness metric provides deterministic, interpretable, zero-latency per-frame expert routing without any neural gating network.
BotSort Track Continuity
Kalman-filter motion prediction maintains animal identity tracks through occlusions and routing switches, providing persistent individual IDs across the full video sequence.
Interactive Gradio Demo with Live Routing Visualization
Deployed application exposes routing decisions, detection overlays, and adjustable Laplacian threshold to make the entire MoE pipeline behavior directly observable.
- [01]Computer Vision
- [02]RT-DETR-X
- [03]YOLO11m
- [04]Mixture of Experts
- [05]Laplacian Variance Router
- [06]BotSort Tracking
- [07]Python
- [08]Gradio
Other Selected Works
Autonomous Drone
A ROS2 autonomous-navigation stack for a DJI Tello: monocular ORB-SLAM3 for real-time visual SLAM with a custom depth-estimation and scale-recovery pipeline, plus occupancy mapping and exploration planning layers in active development.
Learn more →Deep Learning & Medical ImagingConditional Multimodal MRI Synthesis & Brain Tumor Segmentation
A two-model system combining a ResNet U-Net for multi-class brain tumor segmentation with a conditional diffusion model that synthesizes 4-modality MRI scans from masks. Led as a two-person project and trained on TPU.
Learn more →Computer Vision & Edge AIGesture-Controlled Drone
A real-time gesture-recognition pipeline controlling a DJI Tello in flight. Built on a self-collected dataset of 5,000+ hand-landmark samples, a feature-engineered classifier, and temporal smoothing for reliable control.
Learn more →