Hugging Face Daily PapersSara Pieri, Evangelos Kazakos, Shizhe Chen1 min readpaperadvanced
PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection
Summary
PANORAMA introduces a VLM that grounds each phrase of a caption to pixel masks by selecting from a phrase‑conditioned pool of mask proposals, trained jointly with caption generation. It sets a new benchmark on the PanoCaps dataset and outperforms prior methods on several pixel‑level grounding tasks.
- PanoCaps benchmark supplies dense panoptic captions with near‑complete pixel coverage for training and evaluation.
- Defines a phrase‑mask matching protocol and a generalized Panoptic Quality (gPQ) metric that jointly assess text and mask alignment.
- PANORAMA conditions a pretrained segmenter on contextualized phrase embeddings to generate candidate masks and learns to select the correct mask(s) per phrase.
- Joint training of grounding selection with caption generation yields precise entity masks while preserving detailed, mask‑consistent captions, achieving state‑of‑the‑art grounding scores.
Engineers building vision‑language systems that require accurate spatial grounding—for robotics, image editing, or detailed scene understanding—should see this as a practical advancement.
7/10
