Hybrid quantum–classical multimodal fusion under weak cross-modal alignment for bird species recognitionShow others and affiliations
2027 (English)In: Expert systems with applications, ISSN 0957-4174, E-ISSN 1873-6793, Vol. 333, article id 133774Article in journal (Refereed) Published
Abstract [en]
Multimodal learning systems typically assume strong alignment between data modalities, an assumption frequently violated in real-world scenarios such as ecological monitoring, where visual and acoustic observations are weakly or inconsistently paired. This paper proposes a hybrid quantum-classical fusion framework that integrates independently learned modality-specific representations through a Parameterized Quantum Circuit (PQC). The proposed Quantum Fusion Layer (QFL) is designed to capture high-order cross-modal interactions under weak alignment conditions. Experimental evaluation on a multimodal bird species classification task shows that the proposed framework outperforms classical fusion baselines, achieving Accuracy and Macro F1-score of 0.9875 on a held-out test set. Additional analyses comprising stratified cross-validation, ablation studies, robustness evaluation under controlled noise conditions, a parameter-matched classical MLP control, and cross-domain generalization on a geographically out-of-distribution benchmark (Bird-SEA10) of 238 multimodal pairs from Singapore and Thailand indicate stable performance across data partitions, resilience to moderate input degradation, and meaningful generalization under geographic domain shift (cross-domain Accuracy = 0.8445, Macro F1 = 0.8382, ΔF1 = −0.1493 relative to in-domain performance). The parameter-matched MLP control achieves Accuracy of 0.2125 and Macro F1 of 0.0897, suggesting that the observed performance advantage originates from the structural properties of the quantum circuit rather than from reduced parameter capacity. These findings indicate that quantum-enhanced feature transformations may provide an effective mechanism for multimodal fusion under weak alignment, with potential applicability to broader multimodal learning problems involving heterogeneous and imperfectly aligned data sources.
Place, publisher, year, edition, pages
Elsevier Ltd , 2027. Vol. 333, article id 133774
Keywords [en]
Quantum machine learning, Multimodal learning, Multimodal fusion, Weak cross-modal alignment, Ecological monitoring, Bird species recognition
National Category
Computer Sciences
Research subject
Machine Learning
Identifiers
URN: urn:nbn:se:ltu:diva-119406DOI: 10.1016/j.eswa.2026.133774ISI: 001843978000001Scopus ID: 2-s2.0-105046477847OAI: oai:DiVA.org:ltu-119406DiVA, id: diva2:2093020
2026-08-182026-08-182026-08-18Bibliographically approved