GRAIL: Gap-Aware Retrieval for Multimodal Multi-Hop QA

TL;DR GRAIL rewrites retrieval queries inside the embedding space by subtracting already-covered context, steering the search toward the missing evidence. This escapes the "semantic anchoring" trap of additive and LLM-rewriting retrievers, and — combined with a task-adaptive hybrid router — yields a +35.7% macro-averaged gain on MultimodalQA evidence set completion.

Semantic anchoring redundancy trap in MMQA retrieval — **The redundancy trap in MMQA.** Given a table and three emblems, the goal is to fetch the missing visual evidence (Bolton Wanderers' emblem). *Query + Context* stays trapped by the existing Lancashire Senior Cup context (semantic anchoring), and explicit LLM rewriting (IRCoT) prematurely concludes on Manchester City. In contrast, *Query − Context* (GRAIL) implicitly subtracts the covered features and shifts the search toward the unobserved visual target.

The problem: Semantic Anchoring

Multimodal multi-hop QA needs evidence spread across text, tables, and images — and the retriever sets the performance ceiling of the whole pipeline.

To gather multi-hop evidence, modern retrievers iteratively regenerate the query by adding the already-retrieved context. But this additive loop has a structural failure we call Semantic Anchoring: the strong semantic gravity of fetched evidence traps the retriever on prominent entities, fetching redundant assets while pushing the true target out of the top-$K$. Explicit LLM query rewriting fares no better — flattening dense visual features into text tokens triggers semantic drift and heavy latency.

GRAIL: implicit context-subtractive steering

GRAIL operates entirely in a unified multimodal embedding space. Given a query $q$ and a partial evidence set $\mathcal{E}_p$, it aggregates the covered knowledge into a context vector $h_\text{ctx}$ via query-weighted attention, then removes the overlapping component from the query with a learnable gap gate $g$:

$$ q_\text{gap} \;=\; q \;-\; g \cdot \left( \frac{q \cdot h_\text{ctx}}{\lVert h_\text{ctx}\rVert^2} \right) h_\text{ctx}, \qquad g = \sigma\!\left(W_\text{gap}[q; h_\text{ctx}]\right) $$

The refined query $q_\text{gap}$ points toward the unanswered facets of the question. Operating in latent space avoids any linguistic loss, and because every step is differentiable the retrieval signal trains the steering end-to-end. Applied iteratively, it discovers new evidence without accumulating semantic bias.

A two-stage design makes this possible: (1) Unified Semantic Alignment synchronizes text/table/image topologies so cross-modal redundancy can be measured, and (2) the Gap-aware Search Mechanism performs the subtraction above.

Task 1 — Evidence Set Completion

Given a question and a partial gold evidence set, retrieve the single missing piece from the full corpus. We report macro-averaged recall on the MultimodalQA validation split ($q\text{-type} \in \{\text{Compare, Compose, Intersect}\}$). GRAIL and the task-adaptive hybrid beat both the additive baseline and explicit LLM query regeneration — at a fraction of the latency.

Method	R@1	R@5	R@10	R@20	Latency
Query Only	16.24	32.68	40.51	48.54	2–5 ms
Explicit query generation (LLM)
Closed-Book	13.78	26.89	34.65	42.69	0.6–1.0 s
IRCoT	13.42	24.87	30.18	36.21	1.1–1.6 s
Answer-Augmented	16.54	33.24	39.96	48.86	1.2–1.8 s
Implicit vector steering
Additive	11.57	25.89	32.11	39.82	2–5 ms
GRAIL	17.55	33.61	40.81	47.50	3–6 ms
Hybrid (ours)	22.21	39.94	46.90	53.80	3–6 ms

Additive and subtractive steering are complementary: additive helps localized aggregation (Uni-modal, Intersect), while subtractive is essential for compositional, cross-modal reasoning (Compose, Compare). A lightweight self-supervised selector routes each query to the right specialist, giving a +35.7% macro-averaged gain over the additive baseline.

Hybrid framework leveraging complementary specialists — **Best of both worlds.** Routing each query to its optimal specialist yields a best-of-both-worlds result on the MultimodalQA test split.

Task 2 — Sequential Pool Construction

In the open-domain setting there is no partial gold context: the retriever must build the top-$K$ pool from scratch, over several iterative slices. The additive loop degrades rapidly as slices accumulate (error propagation / semantic anchoring), while GRAIL stays stable or improves — turning a fixed budget $K$ into a much deeper effective search.

Set-Recall vs. number of retrieval slices for K=10 and K=20 — **Robust under iterative slicing.** Set-Recall vs. number of retrieval slices for $K=10$ (left) and $K=20$ (right). Additive retrieval collapses as slices increase (−7.11%p / −11.71%p), whereas gap-aware GRAIL stays above the single-shot baseline (+1.76%p / +1.69%p).

Diagnostically, sequential GRAIL surfaces complementary evidence originally ranked around the 30th–35th position into the top-$K$ (high Rank Jump), and keeps a positive Noise-Resilience Margin even when the initial pool contains zero gold items — it self-corrects rather than propagating early mistakes.

BibTeX

@article{o2026grail,
  title   = {Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA},
  author  = {O, Sunah and Lee, Jay-Yoon},
  journal = {arXiv preprint arXiv:2605.28641},
  year    = {2026}
}