Hugging Face Daily PapersAhmadreza Jeddi, Enming Zhang, Jasper GerigkSep 28, 20261 min readpaperadvanced

SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models

Summary

Reasoning vision-language models (VLMs) are expensive due to long visual token sequences. This paper identifies a "representation-utilization gap" where pruned visual tokens are underutilized, not just lost. SCOPD, a self-distillation framework, improves VLM performance with sparse visual contexts, achieving 92.43% of unpruned performance at 10% token retention.

VLM inference is costly due to processing images/videos as long sequences of visual tokens.
Performance degradation from token pruning is partly due to underutilization of remaining information, termed the "representation-utilization gap".
SCOPD uses on-policy self-distillation with a full-context teacher to train a student on pruned visual tokens.
SCOPD+ enhances this by selectively distilling visually sensitive response positions with a small visual-budget intervention.

Engineers and researchers deploying vision-language models will find this method useful for significantly reducing inference costs while maintaining high performance.

8/10

llm machine learning performance computer vision distillation efficiency

Related reading

Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes arxiv.org
Where‑OPD introduces on‑policy self‑distillation for multimodal LLMs where the teacher receives spatially grounded textual guidance from synthetic scenes. Post‑training on these annotation‑free scenes improves real‑world vision‑language benchmarks by an average of 3.23 points.
Hugging Face Daily Papersarxiv.orgSophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc5d1 minpaper
Beyond Selection: Token Parameterization for Extreme Visual Token Compression arxiv.org
This paper introduces Braco, a lightweight coder for extreme visual token compression in vision-language models. It uses a novel token parameterization approach to maintain visual grounding and achieve high accuracy at 23x-64x compression, significantly improving efficiency over prior methods.
Hugging Face Daily Papersarxiv.orgRui Zhong, Yu Li, Zheyu Yan8d1 minpaper
Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning arxiv.org
The paper pinpoints two weaknesses in latent‑reasoning pipelines—suboptimal visual token encoders and deterministic RL sampling—and proposes Scaffolding Minds, which trains a dedicated scaffolding encoder and a stochastic RL sampler (learning both mean and variance). This yields up to +19 points improvement on spatial planning tasks and +5.6 points average gain on nine visual reasoning benchmarks.
Hugging Face Daily Papersarxiv.orgHaoqiang Kang, Yinpeng Chen, Luyang Liu7d1 minpaper
In-Context Robot Learning with VLM Agents arxiv.org
GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…
Hugging Face Daily Papersarxiv.orgDongzhou Cheng, Taoran Yi, Ye Fang20d1 minpaper
PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop arxiv.org
This paper introduces PhysVista, a new benchmark to evaluate the physical intelligence of Vision-Language Models (VLMs) using a closed perception-reasoning-assessment loop. Experiments reveal significant limitations in current VLMs' physical reasoning and plausibility assessment capabilities.
Hugging Face Daily Papersarxiv.orgXinge Peng, Yiting Lu, Tianwu Zhi6d1 minpaper
Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies arxiv.org
Distill frozen world‑model features into a Vision‑Language‑Action policy via a single alignment loss; no teacher at train time, no extra runtime cost. 0.8 B student runs 32 ms / 1.86 GB on RTX 5090, hits 97.9 % on LIBERO and improves RoboCasa‑GR1 from 48.2 % to 50.5 %, with transfer to real single‑arm and bimanual robots.
Hugging Face Daily Papersarxiv.orgTrung Dao, Sankalp Yamsani, Jaden Park15d1 minpaper

SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models

Summary

Related reading

Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenesarxiv.org

Beyond Selection: Token Parameterization for Extreme Visual Token Compressionarxiv.org

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoningarxiv.org

In-Context Robot Learning with VLM Agentsarxiv.org

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Looparxiv.org

Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policiesarxiv.org

Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes arxiv.org

Beyond Selection: Token Parameterization for Extreme Visual Token Compression arxiv.org

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning arxiv.org

In-Context Robot Learning with VLM Agents arxiv.org

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop arxiv.org

Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies arxiv.org