Hugging Face Daily PapersDario PicozziSep 10, 20261 min readpaperadvanced

The information geometry of large language models is shared, learned, and controllable

Summary

The paper shows that the Fisher‑Rao geometry of next‑token probabilities is largely shared across transformer, state‑space and recurrent LLMs, and that this shared geometry can be used to design low‑disturbance interventions that steer model behavior. Experiments demonstrate that geometry predicts semantic transfer, fact acquisition, and enables reusable control better than Euclidean methods.

Output‑space Fisher‑Rao geometry aligns across architectures more strongly than activation‑space geometry.
Shared geometry predicts semantic‑category transfer and correlates with human word‑choice agreement, improving with scale and calibration.
Geometric interventions provide minimum‑disturbance steering, with costs predicted by the geometry and transferable across prompts.
Deeper evidence in pretraining data delays fact acquisition, a pattern consistent across all tested model families.

LLM researchers and engineers should care because the geometric framework offers a principled way to understand and control model behavior without costly retraining.

8/10

llm machine learning transformers control information geometry

Related reading

Register Tokens for Bounded-State Reasoning in Diffusion Language Models arxiv.org
Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.
Hugging Face Daily Papersarxiv.orgAlbert Ge, Chandan Singh, Yufan Zhuang9d1 minpaper
MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads arxiv.org
This paper introduces HEAL, a method to identify and mitigate hallucinations in Multimodal Large Language Models (MLLMs). It finds that hallucinations stem from information distribution drift in "synergy heads" within the attention mechanism and proposes dynamic calibration to steer output towards factual evidence.
Hugging Face Daily Papersarxiv.orgMeng'en Qin, Junye Chen, Jucheng Liu18d1 minpaper
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models arxiv.org
Action tokenizers for autoregressive VLA models often fail to preserve subtle action adjustments, despite good pointwise reconstruction. This paper introduces Physical Rank Consistency (PRC) to measure relational fidelity and ActionPiece, a new tokenizer that uses joint supervision to preserve these physical relationships, significantly improving policy success on robotics benchmarks.
Hugging Face Daily Papersarxiv.orgShijie Lian, Bin Yu, Zhaolong Shen7d1 minpaper
Continual Learning Mechanisms Compose for Long-Horizon Memorization arxiv.org
The paper studies long‑horizon memorization where a language model must learn 100 tasks via continual fine‑tuning without retaining data or task IDs. By composing data, function, and weight anchors with merged LoRA, they boost final retention from 1.2% to 34.9%, a 28× improvement.
Hugging Face Daily Papersarxiv.orgZheyuan Zhang, Alvin Zhang, Daniel Khashabi16d1 minpaperHN2
Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States arxiv.org
Mind2Dialogue introduces a psychology‑guided simulator that generates user mental states and uses privileged distillation to train LLM assistants on Oracle responses that know those states. The resulting models improve personalization and theory‑of‑mind metrics by up to 41 percentage points versus standard instruction‑tuned baselines.
Hugging Face Daily Papersarxiv.orgZixuan Wang, Yufan Zhou, Jinzhou Tang9d2 minpaper
SteerDuplex: Steerable Duplex Speech Dialogue Models arxiv.org
The paper presents SteerDuplex, a full‑duplex speech dialogue model that can be steered along tone, persona, and speed via instruction following, and introduces the SteerBench benchmark to evaluate such steerability. Supervised training yields a 44.5 % pass‑rate lift, and reinforcement‑learning fine‑tuning improves interruption handling and reduces pause barge‑ins, though reward hacking remains a…
Hugging Face Daily Papersarxiv.orgUtkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai12d2 minpaper

The information geometry of large language models is shared, learned, and controllable

Summary

Related reading

Register Tokens for Bounded-State Reasoning in Diffusion Language Modelsarxiv.org

MLLMs Hallucinate when Information Distribution Drifts in Synergy Headsarxiv.org

ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Modelsarxiv.org

Continual Learning Mechanisms Compose for Long-Horizon Memorizationarxiv.org

Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental Statesarxiv.org

SteerDuplex: Steerable Duplex Speech Dialogue Modelsarxiv.org

Register Tokens for Bounded-State Reasoning in Diffusion Language Models arxiv.org

MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads arxiv.org

ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models arxiv.org

Continual Learning Mechanisms Compose for Long-Horizon Memorization arxiv.org

Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States arxiv.org

SteerDuplex: Steerable Duplex Speech Dialogue Models arxiv.org