proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersSimon Schug, Brenden M. Lake1 min readpaperintermediate

Thought without systematicity? Evaluating reasoning models on rule induction tasks

Read on arxiv.org

Summary

The paper builds compositional rule‑induction task families and generates isomorphic variants to probe systematicity in reasoning models. Experiments reveal that models frequently fail on equivalent variants despite solving the original task, indicating a lack of systematic reasoning.

  • Models can solve a rule‑induction task yet often fail on structurally equivalent variants created via recombination or substitution.
  • The authors construct task families with compositional structure and define isomorphisms to test systematicity.
  • Empirical results show a systematicity gap: correct performance on one task does not guarantee performance on its isomorphic counterpart.
  • Systematicity should be incorporated into benchmark suites for reasoning models to assess robust generalization.

Anyone evaluating or developing LLM reasoning abilities needs to know that current models may not generalize systematically across equivalent tasks.

5/10
llmevaluationreasoningsystematicityrule induction

Related reading

  1. Verifiable Social Reasoning for LLM Assistants

    The paper introduces Fuse, a multi‑agent simulation that gives LLM assistants a verifiable ground‑truth task for social reasoning by hiding a target agent’s motive and letting a user‑mediated conversation infer it. Experiments on 12 LLMs show user mediation makes reasoning harder, models are biased by user framing, need more detail than humans, and longer chats don’t always help.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation

    The paper shows that when specialist LLMs are trained only on QA pairs (no explicit reasoning supervision), their optimization implicitly selects a latent distribution of reasoning trajectories. By treating the distilled student as an agnostic probe—since it inherits only the sampled trajectories—the authors empirically demonstrate a strong correlation (across 27 specialist‑student pairs) between…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model

    This paper introduces a novel evaluation framework to assess the physical world reasoning capabilities of omni-modal generative models like MiniMax-H3. It found that MiniMax-H3 achieved an overall success rate of 41.97% across 517 instances, with significant performance variations depending on the input modalities and reasoning tasks.

    Hugging Face Daily Papersarxiv.org2 minpaper

  5. When2Think: Learning Difficulty-Aware Length Control for Efficient Hybrid Reasoning Models

    When2Think introduces a post‑training framework that lets a large reasoning model decide per‑instance how much reasoning depth to allocate, using difficulty‑aware reward shaping (IDAC) and verifier rewards. It cuts token usage by ~28% while boosting Pass@3 by 10% on AIME24 and reaches 40% Pass@3 on AIME25, outperforming compression and routing baselines.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Use Curiosity, Craft, and Care to Decide What AI Should Write

    The post proposes a three‑principle framework—Curiosity, Craft, and Care—to decide how much AI should author each artifact in a software development workflow. It argues that AI can be used aggressively for exploratory, disposable outputs (Curiosity) but should be limited for artifacts that commit the team to decisions (Craft) and for communications that require personal ownership (Care). The auth…

    Atomic Objectatomicobject.com4 min