attention

RSS

1
Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation arxiv.org
Video DeltaNet (VDN) replaces full‑softmax attention in video diffusion models with a hybrid: per‑frame local Softmax for fine detail and a bidirectional linear memory (Video Delta Attention) for long‑range context. A teacher‑alignment schedule injects the linear branch into a pretrained MiniMax H3 model, preserving Softmax for text/audio streams. On eight NVIDIA B200 GPUs VDN‑H3 denoises a 14.3‑…
Hugging Face Daily Papersarxiv.orgHaocheng Xi, Yiming Xie, Hexu Zhao5d1 minpaper
2
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention arxiv.org
VC-Attention introduces a training‑free low‑bit attention pipeline for diffusion transformers. It smooths value tensors via lightweight online clustering (V‑Smooth) and quantizes only the residual after subtracting block means, restoring the mean from the softmax row sum. It also replaces the FP32 softmax exponential with a fused FP8 cast (ExpCast‑FP8) that maps log‑scores directly to E4M3 probab…
Hugging Face Daily Papersarxiv.orgXingyang Li, Dongyun Zou, Shining Zhang8d1 minpaper
3
SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization arxiv.org
This paper introduces SpectralShift, a spectral reparameterization method for extending the context window of Gated DeltaNet (GDN) linear attention models. It reconfigures the decay spectrum by enhancing slow propagation and preserving fast-decaying modes, consistently improving long-context capabilities during continual pretraining.
Hugging Face Daily Papersarxiv.orgZian Liu, Yiwen Hu, Zican Dong9d1 minpaper
4
MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads arxiv.org
This paper introduces HEAL, a method to identify and mitigate hallucinations in Multimodal Large Language Models (MLLMs). It finds that hallucinations stem from information distribution drift in "synergy heads" within the attention mechanism and proposes dynamic calibration to steer output towards factual evidence.
Hugging Face Daily Papersarxiv.orgMeng'en Qin, Junye Chen, Jucheng Liu17d1 minpaper

attention

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generationarxiv.org

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attentionarxiv.org

SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterizationarxiv.org

MLLMs Hallucinate when Information Distribution Drifts in Synergy Headsarxiv.org

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation arxiv.org

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention arxiv.org

SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization arxiv.org

MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads arxiv.org