Structured diagrams such as UML sequence diagrams are a key medium for representing process logic in technical and telecom documentation, and retrieving the correct diagram from a large corpus is a recurring task in industrial deployments. Modern multi-vector vision language retrievers, which compute fine-grained similarities between text tokens of a query and patch tokens of a rendered image, achieve strong performance on document benchmarks but exhibit a specific and consequential failure when applied to sequence diagrams: they are dominated by the textual content of node labels and are nearly blind to the underlying edge topology. Two diagrams that share the same participant labels but encode entirely different message flows collapse to near-identical positions in the embedding space,while the retriever’s response to relabelling a single participant is an order of magnitude larger than its response to deleting a single edge of the same graph-edit distance. 

We address this with StructColQwen, a fine-tuned variant of ColQwen2.5 designed to acquire structural sensitivity without eroding the underlying retriever’s general document retrieval competence. The method has three components. First, we construct training families in which each anchor diagram is paired with a set of structurally perturbed variants generated under precise graph-edit-distance control, supplying the explicit topological supervision ab sent from cross-modal pretraining; a hard-negative sampling strategy keeps the inter-family contrast informative at corpus scale. Second, we combine a standard cross-modal contrastive objective with two structural terms — an intra-family contrastive loss that ranks the anchor above its three perturbations, and a hinge term that prevents an entire family from sliding below the inter-family similarity floor. Third, we fine-tune via two independent LoRA adapters, one on the language model’s self-attention projections and one on the last four blocks of the vision encoder, with all remaining backbone weights frozen. 

On abenchmark built from the public PlantUML sequence-diagram corpus and populated with structural distractors, StructColQwen reaches MRR 0.752 and R@1 0.615 on the distractor pool— surpassing ColQwen’sown officiallyreleased v0.2 retriever(0.334MRR,0.090R@1)and lifting the ColQwen2.5-base backbone it fine-tunes (0.092 MRR, 0.027 R@1) — while retaining nearperfect performance on the distractor-free retrieval setting. An ablation isolates the two structural loss terms as addressing complementary failure modes: the floor hinge restores tail behaviour but cannot improve top-rank accuracy, while the intra-family loss requires the floor hinge to operate on a representation that has not already collapsed. On an internal Ericsson telecom-documentation corpus the gains over v0.2 are consistently larger than on the public benchmark, consistent with the larger diagnostic value that structural cues carry in human authored diagrams.