Hugging Face Daily PapersXingyang Li, Dongyun Zou, Shining Zhang1 min readpaperadvanced
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
Summary
VC-Attention introduces a training‑free low‑bit attention pipeline for diffusion transformers. It smooths value tensors via lightweight online clustering (V‑Smooth) and quantizes only the residual after subtracting block means, restoring the mean from the softmax row sum. It also replaces the FP32 softmax exponential with a fused FP8 cast (ExpCast‑FP8) that maps log‑scores directly to E4M3 probab…
- Value outliers dominate low‑bit attention error; V‑Smooth clusters values per hardware block to improve quantization.
- Quantizing only the residual after block‑mean subtraction lets the mean be recovered from the softmax row sum, avoiding extra passes.
- ExpCast‑FP8 fuses the softmax exponential and FP8 conversion into a single multiply‑add, eliminating the FP32 exponential bottleneck.
- Benchmarks on Wan2.2, LongCat‑Video, HunyuanVideo‑1.5, and MiniMax‑H3 show both higher fidelity than prior low‑bit baselines and substantial speedups (up to 3.6×).
Attention is the main cost in long‑sequence video diffusion models. Low‑bit kernels can cut compute and memory, but quantization errors and the softmax exponential have limited prior solutions. VC‑Attention’s value‑centric smoothing and fused FP8 softmax directly address these bottlenecks, deliveri…
8/10
