RetFormer

Long-document question answering has a cost problem. A vision-language reader like Qwen2.5-VL spends several hundred vision tokens on every page it looks at, so a question over a long report either sees only a few retrieved pages or runs out of context. RetFormer sits between a frozen late-interaction retriever (ColQwen2.5/3) and a frozen reader (Qwen2.5-VL-7B or Qwen3-VL-8B). It compresses each retrieved page’s patch embeddings into K evidence tokens and splices them into the reader’s vision-token slots, so the reader answers from a fraction of the tokens.

How it works

• Retrieval-gated cross-attention. A small set of learned queries cross-attends to a page’s patch embeddings. The retriever’s own MaxSim scores against the question gate that attention, so the tokens the bridge keeps are the ones the retriever already found relevant.
• Question-aware compression. The bridge reads the question as well as the page, in the spirit of InstructBLIP and DocVLM, instead of compressing every page the same way whatever is asked.
• A cross-page consolidator. A second stage merges the evidence from all retrieved pages into a fixed budget of 32 tokens, so the reader’s input no longer grows with the length of the document. By design, that is over a thousand times fewer tokens than the raw pages of a 50-page document.
• Everything else stays frozen. Only the bridge (and, in some variants, a LoRA on the reader) is trained, in two stages: the bridge first learns to match the reader’s own view of a page, then the two are trained together on question answering.

What it measures so far

On MMLongBench-Doc (1,091 questions), token F1:

Compression costs accuracy: with retrieval, RetFormer keeps a little over half of the raw pages’ F1 on a third of the tokens. That trade is only worth making if a learned bridge beats simply throwing tokens away at the same budget, which is what the current work tests.

What went wrong on the way

The second version, a BLIP-2-style two-stage design, trained cleanly on its own losses and then failed completely at generation (F1 0.002 on its own training distribution). Working through it ruled out exposure bias and capacity: the bridge was compressing every page the same way whatever the question, and its output sat off the reader’s visual manifold. The third version fixes both, with question-aware compression and training that ends answers properly, which is what the numbers above use.

Now

The question is now framed as learned against direct compression at a matched budget. At the same number of vision tokens per page, the bridge is compared with naive spatial pooling, token merging (ToMe), and the same bridge with its retrieval gates switched off (a plain Q-Former), on MMLongBench-Doc and LongDocURL, with uncompressed pages as the ceiling.

RetFormer started as a CMU 10-723 project (Spring 2026) with Meltem Sahin Ozkoc, who built the retrieval, precomputation, grounding and long-document data, and Yihao Sun, who built the baselines, the evaluation harness and the failure taxonomy. I built the RetFormer module, its training, the multi-page agent and the analysis of the retrieval gates, and continued the work on my own after the course.