The conducted research aims to create a unified framework for interpreting visual data captured by aerial platforms (UAVs), moving from pixel-level processing to complex semantic and structural representations.
Three-level perception architecture
The system is hierarchically organized to transform raw data into informed decisions:
- Level 1 – Visual Perception: Focuses on extracting fundamental features from RGB images or video streams. It includes dense depth predictors, panoptic segmentation methods (generating semantic and instance masks), and algorithms for object detection in 3D space.
- Level 2 – Neural Scene Reconstruction: Uses static and dynamic neural predictors to generate 3D models. This stage enables the rendering of projective images, depth maps, orthographic views, and Bird’s Eye View (BEV) projections. It also performs localization on Open Street Map (OSM)-based maps.
- Level 3 – Scene Graph: Represents the highest level of abstraction where change detection, road property analysis, and land use detection are performed. It serves as an interface for the decision-making agent.
The ClaraVid synthetic dataset
To train models capable of handling complex scenarios, the ClaraVid dataset was developed, offering major advantages over traditional datasets:
- Multi-View Acquisition: Enables simultaneous capture from multiple viewpoints (High, Mid, Low angle) for high-resolution scenarios.
- Comprehensive Multimodal Annotation: Includes scene-level point clouds (PCL), masks for dynamic objects, panoptic segments (semantic and instance), and accurate depth maps.
- Accessibility: The data is publicly available on the Hugging Face platform: huggingface.co/datasets/radubeche/claravid.
UAVid++ – Advanced semantic segmentation for UAV imagery
Visual perception in aerial (UAV) scenarios poses significant challenges compared to ground-level imagery. Among the main difficulties are:
- extreme scale variations (from large buildings to very small objects such as pedestrians or distant vehicles)
- high density in urban areas
- visual ambiguities caused by observation angles and altitude
Existing datasets, including UAVid, have important limitations:
- coarse annotations (approximate labels)
- semantic inconsistencies between classes
- lack of fine-grained contour details
These limitations directly affect the performance of segmentation models, especially regarding small objects and generalization in new scenarios.
To overcome these limitations, we developed UAVid++, a significantly improved version of the UAVid dataset, which introduces:
High-fidelity annotations
- manual and automatic correction of existing labels
- strict semantic alignment between frames
- much more precise pixel-level contours
Extended semantic taxonomy
- introduction of additional classes relevant to aerial scenarios (e.g., water, sky, rooftops)
- more coherent structure compatible with other UAV datasets
- support for out-of-distribution evaluations
Fine segmentation in complex scenarios
- improving the representation of small and crowded objects
- clearer separation between nearby instances
- reducing ambiguities in dense urban areas
Integration of Large Vision Models (LVM)
To fully leverage the new dataset, a hybrid architecture has been proposed that combines:
- DINO backbone (self-supervised Vision Transformer)
→ provides robust semantic representations and excellent generalization capability - U-Net-inspired head
→ recovers fine spatial details and improves contour precision
This combination addresses a fundamental trade-off in visual perception:
- large models (LVMs) → capture global context well but lose fine-grained details
- specialized models → provide local precision but generalize less effectively
By integrating them, we obtain a model capable of simultaneously handling global context and local detail.
The conducted experiments demonstrate consistent improvements:
- Superior generalization
Models trained on UAVid++ perform robustly across different scenes, altitudes, and camera orientations. - Improved small-object segmentation
Detection and delineation of small-sized objects is significantly more accurate. - Clearer and more stable contours
Object boundaries are better defined, reducing classification errors at borders. - Calitate apropiată de ground truth
Nivelul ridicat al adnotărilor permite utilizarea dataset-ului pentru:- knowledge distillation
- evaluări riguroase ale modelelor
- benchmark-uri realiste
Monocular depth estimation and geo-informed scaling
A major challenge in aerial perception is obtaining accurate metric depth from monocular images. The proposed solution includes:
- UAVid-3D-Scenes: An extension of the UAVid semantic dataset focused on depth, enabling joint research on semantic and 3D tasks.
- Integration of TanDEM-X GDEM: Global elevation data (Digital Elevation Model) is used to ensure correct metric scaling of relative depth models.
- Fluxul de Procesare a Datelor Real-World:
- Alinierea coordonatelor GPS cu imagini satelitare pentru setul UAVid.
- Utilizarea algoritmului Cloth Simulation Filter (CSF) pentru segmentarea eficientă a solului.
- Proiecția și mascarea datelor pentru corelarea precisă între cadrele camerei și modelul digital de teren.
UAVid-3D-Scenes – Depth-oriented extension
UAVid-3D-Scenes is an extension of the UAVid dataset, focused on integrating depth information to enable applications that combine semantic and 3D data. The dataset is publicly available on HuggingFace: huggingface.co/datasets/hrflr/uavid-3d-scenes
The dataset includes:
- dense and sparse depth
- alignment of UAVid frames to global coordinates (using GPS and satellite imagery)
For each frame, data from the TanDEM-X (GDEM) elevation model are integrated, used for terrain information projection.
Additionally, 3D reconstructions from the UFO Depth dataset are used for out-of-distribution evaluations.
The goal is to improve the metric scaling of depth estimates by combining position, altitude, and digital terrain model information.
