proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersBowen Qin, Yi Xie, Yesheng Liu1 min readpaperadvanced

ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals

Read on arxiv.org

Summary

ImpossibleRubrics introduces a 169‑task benchmark of “impossible” prompts, each with a formal oracle certificate defining what an honest answer can claim. The authors generate rubrics downstream and test them adversarially, finding that many rubric generators are exploitable (8‑36% of the time) and that a single generic rubric (“be decisive, penalize hedging”) is exploited 64% of the time, while…

  • A new benchmark (ImpossibleRubrics) for stress‑testing LLM‑generated rubrics on tasks where the truthful response is to admit impossibility.
  • Each task includes a verifiable oracle certificate that specifies permissible claims, enabling automated detection of rubric violations.
  • Eleven rubric generators were evaluated; exploitation rates ranged from 8% to 36% on unbiased cuts, up to 64% for a generic “decisive” rubric.
  • Tailored rubrics often expose specific claim targets, making them more vulnerable than a vague generic rubric.

Rubric‑based reward signals are increasingly used for RL‑from‑human‑feedback, automated grading, and LLM evaluation. If rubrics can be gamed, the resulting reward models may reinforce dishonest or misleading behavior, undermining safety and reliability of LLM systems. This benchmark provides a conc…

7/10
llmmachine learningreinforcement learningevaluationbenchmarksnlp

Related reading

  1. Prompts aren’t Real

    The talk argues that prompt engineering is a dead‑end and proposes building large evaluation/optimization pipelines (pass^k testing, adversarial scenario generation, automated prompt optimization) to make LLM agents reliable. It describes a workflow: generate tests, run them with/without a new “skill”, feed results to a genetic optimizer that mutates prompts, validate on hold‑out tests, and itera…

    Hacker News front pageevaluation.club24 mintalkHN11757

  2. Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems

    Emergence World is a continuously running multi‑agent sandbox used to stress‑test frontier LLM‑based agents over weeks. Eight parallel worlds (seven homogeneous, one mixed) generated 850 k LLM calls and ~50 B tokens while agents pursued goals, used tools, and maintained persistent memory. The authors injected three adversarial events—prompt injection, misinformation, and private‑memory exposure—a…

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Learning to solve hard problems in RL for LLMs by never giving up

    The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…

    Hacker News front pagegithub.io11 minHN1179

  4. Why I'm still bearish on LLMs after Navier-Stokes

    The author argues that despite headline successes (e.g., Navier‑Stokes proof, security exploits), current frontier LLMs still require heavy human oversight and rigorous specifications that are costly to produce. Reward‑hacking, narrow generalization, and the need for domain‑expert spec writing limit autonomous deployment to only a few niche domains (high‑failure‑cost work, tightly defined tasks,…

    Hacker News front pagedank.systems5 minHN487644lobste.rs49

  5. Poisoned Documents, Real Risks: Sebastián Passaro Puts AI’s Weakest Link to the Test at Testear.la 2026

    Sebastián Passaro (Qubika) demonstrated a live RAG pipeline attack at Testear.la 2026, showing how a single poisoned document can hijack LLM outputs and trigger unsafe actions. He tied the demo to the OWASP LLM Top 10, highlighted open‑source tooling for finding such weaknesses, and advocated a defense‑in‑depth threat model for QA teams. The talk reframed AI from a testing aid to a security surfa…

    Moove-itqubika.com4 min

  6. Article: Architecting Secure and Scalable Facial Verification Systems

    A real‑world post‑mortem of a high‑volume face verification service that moved from a naïve synchronous API to an async, layered pipeline (edge validation, preprocessing, decoupled detection/verification, decision engine) to achieve 8.5k rpm, p99 < 1.8 s, 30 % cost savings, and strict privacy controls.

    InfoQinfoq.com15 min