Hugging Face Daily PapersZehao Jin, Ruixuan Deng, Junran WangSep 29, 20261 min readpaperadvanced

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

Summary

The paper shows that standard pretrained transformers stop using deeper layers after only a few lines of context. Adding a rank‑8 LoRA to an early layer while keeping the rest frozen dramatically extends effective reasoning depth, boosting exact accuracy on long‑chain tasks from ~15% to near‑100%.

Pretrained models reliably follow only 1.4–3.6 lines of context, even after extra pretraining loops.
A rank‑8 LoRA added to an early layer (all other weights frozen) extends usable depth to >15 lines on Qwen3‑8B, achieving 99% exact accuracy.
The LoRA creates a relay where middle layers propagate chain identity, letting frozen heads read farther up the chain.
Removing parent‑line attention breaks the relay, confirming its role in extended reasoning.

LLM engineers and researchers should care because a tiny, cheap LoRA edit can unlock far deeper reasoning without full model retraining.

7/10

llm performance ai ml infra lora transformers

Related reading

Disentangling Representation Evolution in Transformers through Directional Decomposition arxiv.org
The paper decomposes transformer representation updates into parallel and perpendicular components, showing that parallel updates beyond the residual path are significant. Suppressing parallel updates during pretraining improves validation loss and downstream metrics.
Hugging Face Daily Papersarxiv.orgShwai He, Haichao Zhang, Shen Yan22d1 minpaper
Continual Learning Mechanisms Compose for Long-Horizon Memorization arxiv.org
The paper studies long‑horizon memorization where a language model must learn 100 tasks via continual fine‑tuning without retaining data or task IDs. By composing data, function, and weight anchors with merged LoRA, they boost final retention from 1.2% to 34.9%, a 28× improvement.
Hugging Face Daily Papersarxiv.orgZheyuan Zhang, Alvin Zhang, Daniel Khashabi29d1 minpaperHN2
Attention Is All You Need arxiv.org
The paper proposes the Transformer, a sequence‑to‑sequence model that relies solely on self‑attention, eliminating recurrence and convolutions. It achieves state‑of‑the‑art translation BLEU scores while training orders of magnitude faster.
Hall of Famearxiv.orgAshish Vaswani et al.10y27 minpaper
Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models arxiv.org
Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.
Hugging Face Daily Papersarxiv.orgJianman Lin, Shailesh Shailesh, Zhongyi Luo25d1 minpaper
Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs arxiv.org
The paper demonstrates that transformer LLMs exhibit a linear superposition property where combined inputs produce a blended next‑token distribution, and that lightweight fine‑tuning can restore this linearity. It also introduces a guided decoding algorithm that extracts two distinct, coherent continuations from one forward pass.
Hugging Face Daily Papersarxiv.orgPavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk12d1 minpaper
Transformers now runs llama.cpp quants huggingface.co
Transformers now supports GGUF quantized checkpoints (used by llama.cpp) via the `kernels` library. Load a GGUF model with `from_pretrained` on Apple Silicon, run generation with standard Transformers APIs, or serve it via `transformers serve`. Benchmarks on an M2 Max show token‑throughput comparable to llama.cpp, with fallback to dequantization if kernels are missing. The integration also enable…
Hugging Facehuggingface.coMarc Sun, Arthur Zucker, Lysandre14d10 minHN4

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

Summary

Related reading

Disentangling Representation Evolution in Transformers through Directional Decompositionarxiv.org

Continual Learning Mechanisms Compose for Long-Horizon Memorizationarxiv.org

Attention Is All You Needarxiv.org

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Modelsarxiv.org

Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMsarxiv.org

Transformers now runs llama.cpp quantshuggingface.co

Disentangling Representation Evolution in Transformers through Directional Decomposition arxiv.org

Continual Learning Mechanisms Compose for Long-Horizon Memorization arxiv.org

Attention Is All You Need arxiv.org

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models arxiv.org

Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs arxiv.org

Transformers now runs llama.cpp quants huggingface.co