Hugging Face Daily PapersHaoqiang Kang, Yinpeng Chen, Luyang LiuSep 29, 20261 min readpaperadvanced

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning

Summary

The paper pinpoints two weaknesses in latent‑reasoning pipelines—suboptimal visual token encoders and deterministic RL sampling—and proposes Scaffolding Minds, which trains a dedicated scaffolding encoder and a stochastic RL sampler (learning both mean and variance). This yields up to +19 points improvement on spatial planning tasks and +5.6 points average gain on nine visual reasoning benchmarks.

A dedicated scaffolding encoder replaces off‑the‑shelf vision encoders to produce task‑aligned latent visual tokens.
RL stage learns both mean and variance of the latent sampler, enabling stochastic exploration of latent trajectories.
Combined improvements raise FrozenLake spatial planning scores by +9.5 (up to +19 on 32×32 grids).
Average gain of +5.6 points across nine visual‑centric reasoning benchmarks demonstrates broader applicability.

Researchers and engineers building multimodal vision‑language models should care because better latent representations and stochastic RL sampling directly boost reasoning performance.

7/10

machine learning reinforcement learning multimodal computer vision latent representation

Related reading

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering arxiv.org
Imagine3D-LLM augments a multimodal LLM with a small set of learnable summary tokens that are decoded into a compact 3D Gaussian splatting of the scene, supervised by photometric reconstruction. This auxiliary task improves cross‑view reasoning and yields better results on spatial‑reasoning benchmarks.
Hugging Face Daily Papersarxiv.orgJaewoo Jung, Hyeonseo Yu, Honggyu An6d1 minpaper
Reasoning with Image Generation arxiv.org
ReImaGin proposes using image generation models as a flexible visual reasoning mechanism for multimodal LLMs, moving beyond rigid visual tools. It consistently outperforms text-only and specialist vision-tool baselines by up to 25% on diverse visual reasoning tasks.
Hugging Face Daily Papersarxiv.orgNishad Singhi, Hector Garcia Rodriguez, Aditya Arora20d1 minpaper
PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop arxiv.org
This paper introduces PhysVista, a new benchmark to evaluate the physical intelligence of Vision-Language Models (VLMs) using a closed perception-reasoning-assessment loop. Experiments reveal significant limitations in current VLMs' physical reasoning and plausibility assessment capabilities.
Hugging Face Daily Papersarxiv.orgXinge Peng, Yiting Lu, Tianwu Zhi5d1 minpaper
SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models arxiv.org
Reasoning vision-language models (VLMs) are expensive due to long visual token sequences. This paper identifies a "representation-utilization gap" where pruned visual tokens are underutilized, not just lost. SCOPD, a self-distillation framework, improves VLM performance with sparse visual contexts, achieving 92.43% of unpruned performance at 10% token retention.
Hugging Face Daily Papersarxiv.orgAhmadreza Jeddi, Enming Zhang, Jasper Gerigk7d1 minpaper
Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models arxiv.org
Latent-Foresight is an end-to-end framework that jointly learns a latent tokenizer and a flow-based generative dynamics model for world modeling. It explicitly shapes representations for temporal predictability, outperforming two-stage baselines in future scene understanding tasks.
Hugging Face Daily Papersarxiv.orgEfstathios Karypidis, Spyros Gidaris, Nikos Komodakis4d1 minpaper
Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models arxiv.org
Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.
Hugging Face Daily Papersarxiv.orgJianman Lin, Shailesh Shailesh, Zhongyi Luo24d1 minpaper

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning

Summary

Related reading

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answeringarxiv.org

Reasoning with Image Generationarxiv.org

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Looparxiv.org

SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Modelsarxiv.org

Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Modelsarxiv.org

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Modelsarxiv.org

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering arxiv.org

Reasoning with Image Generation arxiv.org

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop arxiv.org

SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models arxiv.org

Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models arxiv.org

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models arxiv.org