Hugging Face Daily PapersYueyan Li, Haibo Wang, Caixia YuanSep 20, 20261 min readpaperadvanced

Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms

Summary

The paper investigates why video diffusion models often break physical laws, pinpointing excessive spatial decay from Rotary Position Embedding (RoPE) as the culprit. By analyzing cross‑attention trajectories and self‑attention patterns, the authors identify specific attention heads that drive motion planning. They propose a lightweight fix: scaling RoPE frequency per denoising step, which empiri…

Early denoising stages form motion trajectories; a 'first shape, then details' pattern emerges.
A small subset of cross‑attention heads dominates motion planning.
RoPE causes rapid spatial attention decay, locking candidate regions into implausible positions.
Scaling RoPE frequency across denoising steps mitigates decay, allowing better motion exploration.

Physical consistency is a key barrier for deploying text‑to‑video diffusion models in real‑world applications (e.g., simulation, education, content creation). Understanding and fixing the internal attention dynamics offers a path to more reliable generative video systems without costly data curatio…

8/10

machine learning computer vision attention rope video diffusion

Related reading

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation arxiv.org
Video DeltaNet (VDN) replaces full‑softmax attention in video diffusion models with a hybrid: per‑frame local Softmax for fine detail and a bidirectional linear memory (Video Delta Attention) for long‑range context. A teacher‑alignment schedule injects the linear branch into a pretrained MiniMax H3 model, preserving Softmax for text/audio streams. On eight NVIDIA B200 GPUs VDN‑H3 denoises a 14.3‑…
Hugging Face Daily Papersarxiv.orgHaocheng Xi, Yiming Xie, Hexu Zhao5d1 minpaper
Learning Foresight without Explicit Trajectories for 3D Diffusion Policies arxiv.org
This paper introduces Movement Trend Guidance (MTG), a method to provide foresight to 3D diffusion policies for robotic manipulation without explicit trajectory planning. MTG learns a compact latent representation of interaction evolution, significantly improving performance on various benchmarks with minimal parameter overhead.
Hugging Face Daily Papersarxiv.orgZhongbo Zhang, Zaibin Zhang, Yifan Wang5d1 minpaper
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention arxiv.org
VC-Attention introduces a training‑free low‑bit attention pipeline for diffusion transformers. It smooths value tensors via lightweight online clustering (V‑Smooth) and quantizes only the residual after subtracting block means, restoring the mean from the softmax row sum. It also replaces the FP32 softmax exponential with a fused FP8 cast (ExpCast‑FP8) that maps log‑scores directly to E4M3 probab…
Hugging Face Daily Papersarxiv.orgXingyang Li, Dongyun Zou, Shining Zhang8d1 minpaper
PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control arxiv.org
PhysStream introduces a two‑stage autoregressive video generator that uses online‑derived positional and tracking maps (structured scene memory) and sparse velocity‑increment signals to enable fine‑grained, physics‑grounded control of multi‑object tabletop scenes. It cuts motion distribution error by 33 % and trajectory error by 12 % versus strong baselines, and wins 85 % of human preference test…
Hugging Face Daily Papersarxiv.orgChuhao Chen, Peter Wonka, Chaoyang Wang7d1 minpaper
DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models apple.com
DACA‑GRPO adds denoising‑aware credit assignment to GRPO‑style RL trainers for diffusion LLMs. It computes per‑token importance scores from intermediate denoising steps and uses stratified masking to reduce mean‑field bias in likelihood estimates. Plug‑and‑play on three existing GRPO methods, it yields consistent gains on seven downstream tasks (up to +5.6 pp math, +7.4 pp code, +36.3 pp constrai…
Apple Machine Learning Researchapple.com6d1 minpaper
WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory arxiv.org
WorldCrafter introduces a camera‑queryable implicit 3D‑aware memory that compresses multi‑view evidence into a fixed token budget for video generation, improving long‑horizon consistency and camera‑control accuracy in streamed scene exploration.
Hugging Face Daily Papersarxiv.orgWangbo Yu, Kunhao Liu, Wenbo Hu1d1 minpaper

Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms

Summary

Related reading

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generationarxiv.org

Learning Foresight without Explicit Trajectories for 3D Diffusion Policiesarxiv.org

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attentionarxiv.org

PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Controlarxiv.org

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Modelsapple.com

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memoryarxiv.org

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation arxiv.org

Learning Foresight without Explicit Trajectories for 3D Diffusion Policies arxiv.org

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention arxiv.org

PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control arxiv.org

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models apple.com

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory arxiv.org