proomt

Search

Search posts, papers, and topics

All posts

Hacker News front page24 min readtalkintermediate

Prompts aren’t Real

Read on evaluation.clubHN 117·57 comments

Summary

The talk argues that prompt engineering is a dead‑end and proposes building large evaluation/optimization pipelines (pass^k testing, adversarial scenario generation, automated prompt optimization) to make LLM agents reliable. It describes a workflow: generate tests, run them with/without a new “skill”, feed results to a genetic optimizer that mutates prompts, validate on hold‑out tests, and itera…

  • Prompt text is treated as a fragile artifact; instead focus on systematic evaluation of agent behavior.
  • Use large‑scale pass^k tests (repeated runs) to detect flaky LLM outputs.
  • Generate adversarial and benign scenarios (often via another LLM) to form a test suite.
  • Automate prompt refinement with an optimizer (e.g., genetic algorithm) that iteratively mutates prompts based on test scores.

As LLM‑driven agents move from research demos to production services, reliability becomes a core concern. The talk highlights a systematic, test‑driven approach to mitigate prompt brittleness, which is relevant for teams building AI‑powered products that must meet safety, brand, and functional guar…

4/10
devopsllmoptimizationevaluationagentsprompt engineering

Related reading

  1. Learning to solve hard problems in RL for LLMs by never giving up

    The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…

    Hacker News front pagegithub.io11 minHN1179

  2. Agentic Skill Decay

    Addy Osmani warns that AI agents can short‑circuit the hands‑on practice (“reps”) that builds deep expertise and judgment. He recommends deliberately inserting hypothesis‑forming, “why” questioning, diff inspection, failure prediction, and occasional manual coding into the workflow, especially for junior engineers. A 2026 Anthropic study showed junior developers using AI scored 17 % lower on a fo…

    Addy Osmaniaddyosmani.com16 min

  3. ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals

    ImpossibleRubrics introduces a 169‑task benchmark of “impossible” prompts, each with a formal oracle certificate defining what an honest answer can claim. The authors generate rubrics downstream and test them adversarially, finding that many rubric generators are exploitable (8‑36% of the time) and that a single generic rubric (“be decisive, penalize hedging”) is exploited 64% of the time, while…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems

    Emergence World is a continuously running multi‑agent sandbox used to stress‑test frontier LLM‑based agents over weeks. Eight parallel worlds (seven homogeneous, one mixed) generated 850 k LLM calls and ~50 B tokens while agents pursued goals, used tools, and maintained persistent memory. The authors injected three adversarial events—prompt injection, misinformation, and private‑memory exposure—a…

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization

    COBRA‑Skills uses a contextual‑bandit loop to selectively evaluate and evolve LLM agent skills, achieving better performance with roughly half the evaluation cost of prior methods. The framework works with limited examples and remains robust across different agent setups and even when the target model creates its own skills.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Changing the game: How Google uses agentic AI to secure hundreds of millions of lines of code

    Google’s AI & Infrastructure team built an agentic pipeline (Mantis) that runs pre‑submit AI‑driven scans on every code check‑in, validates findings with a fast triage agent (AST + call‑graph analysis) achieving >92% precision in <1 min, then auto‑generates fixes via a bug‑fix agent. Localized threat models and a two‑step scan cut false‑positives to ~3% and prevent hundreds of vulnerabilities eac…

    Google Cloud Bloggoogle.com4 min