Hugging Face Daily PapersMohamed Eltahir, Fardows Adam, Duaa M. TahirSep 28, 20261 min readpaperadvanced

AnswerMap: Faithful Spatial Interpretability of VLMs from Answer Posteriors

Summary

AnswerMap is a novel, training-free, black-box method for generating faithful spatial interpretability maps for Vision-Language Models (VLMs) directly from their output posteriors. It queries the VLM with image bands and yes/no relevance questions, demonstrating higher faithfulness than attention maps and enabling new applications like object localization and hallucination detection.

AnswerMap generates spatial interpretability maps for VLMs using only the output head, making it black-box.
It is training-free and task-agnostic, constructed by querying the VLM with image bands and yes/no questions.
Validation shows AnswerMap is more faithful than attention maps, flipping 53% of correct answers upon region deletion vs 19% for attention.
It enables native derivation of continuous outputs, bypassing reliance on discrete text tokens for tasks like localization.

This paper offers a novel, black-box approach to VLM interpretability and control, which is crucial for understanding and improving the reliability of these complex models in real-world applications.

8/10

llm machine learning computer vision interpretability vlm black box

Related reading

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop arxiv.org
This paper introduces PhysVista, a new benchmark to evaluate the physical intelligence of Vision-Language Models (VLMs) using a closed perception-reasoning-assessment loop. Experiments reveal significant limitations in current VLMs' physical reasoning and plausibility assessment capabilities.
Hugging Face Daily Papersarxiv.orgXinge Peng, Yiting Lu, Tianwu Zhi6d1 minpaper
Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering arxiv.org
Imagine3D-LLM augments a multimodal LLM with a small set of learnable summary tokens that are decoded into a compact 3D Gaussian splatting of the scene, supervised by photometric reconstruction. This auxiliary task improves cross‑view reasoning and yields better results on spatial‑reasoning benchmarks.
Hugging Face Daily Papersarxiv.orgJaewoo Jung, Hyeonseo Yu, Honggyu An7d1 minpaper
LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception arxiv.org
LEAP is a framework for long audio-video question answering that addresses context limits by retrieving relevant evidence blocks. It uses a two-stage process of block-wise localization and a bounded answer pass, achieving significant performance improvements on AVQA benchmarks.
Hugging Face Daily Papersarxiv.orgJuyi Lin, Zhiqiang Lao, Jiali Cui6d1 minpaper
SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models arxiv.org
Reasoning vision-language models (VLMs) are expensive due to long visual token sequences. This paper identifies a "representation-utilization gap" where pruned visual tokens are underutilized, not just lost. SCOPD, a self-distillation framework, improves VLM performance with sparse visual contexts, achieving 92.43% of unpruned performance at 10% token retention.
Hugging Face Daily Papersarxiv.orgAhmadreza Jeddi, Enming Zhang, Jasper Gerigk8d1 minpaper
Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning arxiv.org
The paper pinpoints two weaknesses in latent‑reasoning pipelines—suboptimal visual token encoders and deterministic RL sampling—and proposes Scaffolding Minds, which trains a dedicated scaffolding encoder and a stochastic RL sampler (learning both mean and variance). This yields up to +19 points improvement on spatial planning tasks and +5.6 points average gain on nine visual reasoning benchmarks.
Hugging Face Daily Papersarxiv.orgHaoqiang Kang, Yinpeng Chen, Luyang Liu7d1 minpaper
PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection arxiv.org
PANORAMA introduces a VLM that grounds each phrase of a caption to pixel masks by selecting from a phrase‑conditioned pool of mask proposals, trained jointly with caption generation. It sets a new benchmark on the PanoCaps dataset and outperforms prior methods on several pixel‑level grounding tasks.
Hugging Face Daily Papersarxiv.orgSara Pieri, Evangelos Kazakos, Shizhe Chen20d1 minpaper

AnswerMap: Faithful Spatial Interpretability of VLMs from Answer Posteriors

Summary

Related reading

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Looparxiv.org

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answeringarxiv.org

LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perceptionarxiv.org

SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Modelsarxiv.org

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoningarxiv.org

PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selectionarxiv.org

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop arxiv.org

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering arxiv.org

LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception arxiv.org

SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models arxiv.org

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning arxiv.org

PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection arxiv.org