Introduction: In mission-critical scenarios, such as firefighting, real-time updates are crucial for safety. Fire, smoke, moving obstacles, and other dangers can make commonly used techniques ineffective. This problem creates a need for a robust, multimodal framework to support navigation systems under transmission latency and bandwidth constraints.

Research Question: How can robust obstacle detection and reporting be achieved in visually degraded environments by effectively leveraging complementary sensing and depth estimation techniques?

Method: Following the Design Science Research Methodology, an end-to-end framework was designed to operate in visually degraded environments. It is divided into two main parts, located separately on a portable device and an edge server. The portable device includes a novel task-aware semantic encoder for stereo thermal depth estimation, which is later performed on an edge server, together with object detection, cross-modal comparison between LiDAR and estimated depth maps, and obstacle localization and confidence scoring, also performed on the edge server. The system was trained and evaluated on a dataset consisting of synchronized stereo thermal images and LiDAR depth maps, collected in urban and forest settings, and including smoke and fire.

Results: The task-aware encoder can achieve up to 27x compression of stereo thermal images without introducing any degradation in depth estimation (end-point error of 4.366 vs 4.391 baseline) nor object detection and localization, and up to 64x compression with only a modest performance loss in depth estimation (end-point error of 4.69). The introduced encoding technique allows for a significant reduction in transmission latency when low bandwidth is available (95% latency reduction at 1 Mbps with 27x compression rate). Clustering-based object detection on LiDAR depth map results in an average F2 score of 0.572, and on estimated depth maps, in an average F2 score of 0.408. Filtering out disagreeing detections through cross-modal comparison results in significant improvements in precision and significant degradation in recall.

Discussion: The framework shows that task-aware encoding can considerably improve transmission of stereo thermal images over bandwidth constrained links for depth estimation, object detection, and localization. Moreover, cross-modal comparison provides a mechanism for scoring detection trustworthiness, and the framework can still operate (although on reduced quality) when one modality becomes unavailable. However, its performance in fire and smoke environments remains unclear and requires further work.