Gesture-Controlled Drone
Real-Time 3D Hand Gesture Recognition for Live DJI Tello Quadrotor Flight Control
Project Domain
Computer Vision & Edge AI
Execution Stage
Interactive Prototype
Timeline / Year
2026 · Real-Time Vision System
Repository & Code
GitHub Repository →Visual Documentation & Artifacts
Click any photo to enlargeEngineering with real-world constraints.
This project implements a complete touchless control interface for a DJI Tello quadrotor using only static hand gestures captured through a standard webcam, without any specialized depth sensors or data gloves.
The fundamental design decision was to avoid feeding raw RGB image frames into a heavy convolutional neural network. Instead, the pipeline extracts structured geometric hand representations — 21 three-dimensional landmarks from MediaPipe Hands, normalized for hand size and camera distance, augmented with 19 invariant joint angles — and classifies these compact feature vectors with a lightweight dense network. This approach is fast, generalizes across individuals, and requires no GPU for real-time inference.
The dataset was independently collected and carefully curated across 8 distinct individuals to ensure the model generalizes across hand sizes, skin tones, and personal gesture variations rather than overfitting to a single person's hand proportions.
Design Approach & Feature Engineering Philosophy
Controlling a drone with camera-based gestures requires reliable, consistent predictions even as hands shift position, change distance from the camera, appear at different angles, or transition between gestures. A model that works perfectly for one person but fails for another is not a usable control interface.
Rather than training a convolutional model to recognize gestures directly from image pixels — which would require a massive dataset covering all possible hand appearances, lighting conditions, and backgrounds — we extract an abstract geometric representation from MediaPipe Hands that is inherently invariant to many of these confounds.
The 82-dimensional feature vector encodes the geometric shape of the hand rather than its visual appearance: 63 normalized coordinate values describe where each joint is relative to the wrist and palm width, and 19 computed angle values describe how each finger is bent and spread. These features remain consistent across individuals and camera distances, allowing a simple dense network to learn gesture boundaries on a dataset that is feasible to collect manually.
Component Deep Dives
Standard Webcam Video Stream
│
▼
MediaPipe Hands (21 3D Landmarks)
│
▼
Geometric Feature Extraction
(63 Normalized Coords + 19 Invariant Angles)
│
▼
Dense Neural Classifier (82-Dim Input)
│
▼
15-Frame Temporal Majority Vote
│
▼
DJITelloPy Flight CommandsLandmark Extraction & Normalization
MediaPipe Hands detects 21 three-dimensional hand landmarks per frame: one per finger joint (4 per finger × 5 fingers = 20 joints) plus the wrist. Each landmark is represented as an (x, y, z) coordinate in image-space units.
Raw image-space coordinates are useless for classification because they encode absolute hand position rather than gesture shape. A hand in the top-left of the frame and the same hand in the bottom-right produce completely different landmark coordinates despite performing the same gesture.
The normalization step re-centers the coordinate system at the wrist landmark (subtracting wrist position from all points) and scales by palm width (the distance between wrist and middle finger base joint). This produces a representation where all coordinates are relative to the hand's own geometry, independent of where it appears in the frame and how far it is from the camera. Left-hand landmarks are mirrored to match right-hand training data.
Explore Full Codebase on GitHub
Includes complete source code, installation instructions, launch files, and documentation.
Quantitative Metrics
5,259
Dataset Size
Labeled hand landmark samples
8 People
Contributors
Varied hand sizes, skin tones, and gesture styles
82 Dims
Feature Vector
63 normalized coordinates + 19 invariant joint angles
15 Frames
Smoothing Window
Temporal majority vote preventing spurious commands
Key Capabilities
MediaPipe 3D Landmark Extraction
Extracts 21 3D hand landmark coordinates per frame, centered on the wrist and scaled by palm width to produce position- and scale-invariant hand geometry.
Invariant Joint Angle Engineering
Computes 15 finger-bend angles and 4 finger-spread angles that remain consistent across camera viewpoint changes and individual hand proportion differences.
Lightweight Dense Neural Classifier
128→64→23-class dense network with dropout trained on the 82-dimensional feature vector, achieving real-time inference speeds without GPU requirements.
15-Frame Majority Vote Buffer
Temporal smoothing over a rolling 15-frame prediction window filters out single-frame misclassifications caused by motion blur or brief tracking instability.
Hold-to-Confirm Flight Safety
Gestures must be held steadily for a minimum window before issuing commands, preventing accidental triggering during transitions between gestures.
Multi-Individual Dataset Collection
5,259 labeled samples from 8 diverse contributors, explicitly targeting generalization across hand sizes, skin tones, and personal gesture execution styles.
- [01]Computer Vision
- [02]MediaPipe Hands
- [03]TensorFlow
- [04]Python
- [05]Feature Engineering
- [06]Temporal Smoothing
- [07]DJI Tello
- [08]DJITelloPy
Other Selected Works
Autonomous Drone
A ROS2 autonomous-navigation stack for a DJI Tello: monocular ORB-SLAM3 for real-time visual SLAM with a custom depth-estimation and scale-recovery pipeline, plus occupancy mapping and exploration planning layers in active development.
Learn more →Deep Learning & Medical ImagingConditional Multimodal MRI Synthesis & Brain Tumor Segmentation
A two-model system combining a ResNet U-Net for multi-class brain tumor segmentation with a conditional diffusion model that synthesizes 4-modality MRI scans from masks. Led as a two-person project and trained on TPU.
Learn more →Robotics & Embedded SystemsGesture-Controlled Robotic Hand
An MPU6050-driven glove that maps hand orientation to real-time servo control on a 4-DOF robotic arm — connecting embedded sensing directly to actuation.
Learn more →