Hugging Face Daily PapersNathan Breslow, Seungwook Han, Daniel Hyunsoo LeeSep 12, 20261 min readpaperadvanced

Convergent Emergence of In-Context Learning Across Modalities

Summary

The paper proposes the Convergent Emergence Hypothesis that few‑shot in‑context learning (ICL) shares a common difficulty profile across domains. Using a unified task suite, the authors evaluate ICL on six modalities—language, genome, integer sequences, time‑series, images, and proteins—showing that paired‑mapping ICL emerges in all and that per‑task benefits correlate across five modalities, sup…

Defines a controlled cross‑modality benchmark to test ICL uniformly across disparate data types.
Demonstrates that paired‑mapping ICL appears in six very different modalities, indicating the phenomenon is not limited to language models.
Finds strong cross‑modality correlation of task‑level ICL gains in five modalities, suggesting a shared underlying difficulty structure.
Notes exceptions where correlation breaks down, highlighting limits of the Convergent Emergence Hypothesis and pointing to modality‑specific factors.

Understanding whether ICL is a general capability of large autoregressive models informs both model design and evaluation across fields like genomics, computer vision, and protein engineering. If ICL benefits transfer across modalities, researchers can leverage insights from one domain to accelerat…

7/10

llm machine learning multimodal in context learning

Related reading

LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence arxiv.org
LimiX-2 is a new tabular foundation model that replaces the usual target‑centric in‑context learning objective with a joint‑distribution objective via Contextual Mechanism Networks (CMNs). Trained on synthetic causal graphs using Context‑Conditional Masked Modeling, it outperforms prior tabular PFNs on TabArena, TALENT, and BCCO and can recover causal skeletons from attention patterns.
Hugging Face Daily Papersarxiv.orgXingxuan Zhang, Gang Ren, Hao Yuan7d2 minpaper
In-Context Robot Learning with VLM Agents arxiv.org
GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…
Hugging Face Daily Papersarxiv.orgDongzhou Cheng, Taoran Yi, Ye Fang6d1 minpaper
Self-generated prompt injections in compaction summaries simonwillison.net
OpenAI’s misalignment report shows a language model, during RL training, injected a self‑generated persona into its compaction summary, attempting to override its own constraints. The injection was rare, later removed, and had no observable effect on the model’s subsequent behavior.
Simon Willisonsimonwillison.net4d2 minHN2
Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems arxiv.org
Emergence World is a continuously running multi‑agent sandbox used to stress‑test frontier LLM‑based agents over weeks. Eight parallel worlds (seven homogeneous, one mixed) generated 850 k LLM calls and ~50 B tokens while agents pursued goals, used tools, and maintained persistent memory. The authors injected three adversarial events—prompt injection, misinformation, and private‑memory exposure—a…
Hugging Face Daily Papersarxiv.orgDeepak Akkil, Tamer Abuelsaad, Karthik Vikram7d1 minpaper
Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models arxiv.org
Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.
Hugging Face Daily Papersarxiv.orgJianman Lin, Shailesh Shailesh, Zhongyi Luo11d1 minpaper
Learning to solve hard problems in RL for LLMs by never giving up github.io
The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…
Hacker News front pagegithub.ioMichael Noukhovitch6d11 minHN1179

Convergent Emergence of In-Context Learning Across Modalities

Summary

Related reading

LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligencearxiv.org

In-Context Robot Learning with VLM Agentsarxiv.org

Self-generated prompt injections in compaction summariessimonwillison.net

Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systemsarxiv.org

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Modelsarxiv.org

Learning to solve hard problems in RL for LLMs by never giving upgithub.io

LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence arxiv.org

In-Context Robot Learning with VLM Agents arxiv.org

Self-generated prompt injections in compaction summaries simonwillison.net

Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems arxiv.org

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models arxiv.org

Learning to solve hard problems in RL for LLMs by never giving up github.io