Hugging Face Daily PapersGuangyu Sun, Shlok Kumar Mishra, Wentao Bao1 min readpaperadvanced
FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation
Summary
FLAT is a pre‑training framework that encodes images and text into a shared 1D token sequence with variable length via nested dropout, enabling the same embeddings for cross‑modal retrieval and generation. It attains state‑of‑the‑art scores on COCO and Flickr30K for captioning, retrieval, and text‑to‑image generation, and supports interpolation and zero‑shot composed retrieval.
- FLAT maps visual and textual inputs into a unified 1D token space, using nested dropout to allow flexible prefix lengths.
- Joint training combines contrastive alignment with bidirectional generative decoders, so embeddings serve both discriminative and generative purposes.
- Achieves competitive results: 71.1 GenEval (pre‑train) and 83.1 (fine‑tuned) on T2I, 40.5 BLEU‑4 & 138.6 CIDEr on COCO captioning, Recall@5 86.8/75.8 (I2T/T2I) on COCO.
- Representations support linear interpolation, latent space arithmetic, and zero‑shot composed retrieval.
Engineers building multimodal systems will care because FLAT unifies retrieval and generation in one flexible representation, simplifying pipelines and improving performance.
7/10
