proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersCaiqi Zhang, Xiaochen Zhu, Chengzu Li2 min readpaperadvanced

Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents

Read on arxiv.org

Summary

XConf (eXperiential Confidence) is a novel method for estimating language model confidence by leveraging the model's accumulated experience from past graded episodes. It significantly outperforms existing methods like self-consistency in discrimination and calibration, at a fraction of the computational cost, across various tasks.

  • XConf estimates LLM confidence by recalling past similar tasks and their outcomes, then having the model reflect on its own failure modes.
  • The method is format-general, requires no logit access or weight updates, and adds only one answer generation cost.
  • It beats or matches ten-sample self-consistency in AUROC on 23 of 24 comparisons and has much lower calibration error (ECE).
  • XConf achieves these improvements at a tenth of the generation cost compared to ten-sample self-consistency.

This paper introduces a practical and efficient paradigm for improving the trustworthiness and reliability of LLM deployments by providing more accurate and calibrated confidence estimates.

8/10
llmmachine learningevaluationml infraagentsconfidence estimation

Related reading

  1. PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?

    PACT is a new benchmark designed to measure how well enterprise LLM agents follow compliance rules, especially when under user pressure. It found substantial variability across 22 models, with even the strongest assistants misapplying rules 6-10% of the time, and user pressure increasing violation rates by 65% on average.

    Hugging Face Daily Papersarxiv.org2 minpaper

  3. Article: Architecting Secure and Scalable Facial Verification Systems

    A real‑world post‑mortem of a high‑volume face verification service that moved from a naïve synchronous API to an async, layered pipeline (edge validation, preprocessing, decoupled detection/verification, decision engine) to achieve 8.5k rpm, p99 < 1.8 s, 30 % cost savings, and strict privacy controls.

    InfoQinfoq.com15 min

  4. Presentation: Teaching Engineers, Trusting AI: How Education Enabled Autonomous Code Review

    Duolingo’s DevEx AI team built a program of AI‑literacy workshops, observability dashboards, office‑hours, and vendor partnerships to get engineers comfortable with LLM‑based tools. With that foundation they launched a PR‑risk‑assessment bot that auto‑approves low‑risk pull requests, cutting review bottlenecks while keeping defect rates flat.

    InfoQinfoq.com24 mintalk

  5. A New Framework for Open Source AI

    Mozilla and partners published a paper proposing a layered, gradient openness framework for foundation models, defining openness for data, code, weights, docs, and deployment. The framework gives developers, regulators, and civil society a common language to evaluate openness and safety beyond a binary label.

    Mozilla Automation Teammozilla.org3 min

  6. Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States

    Mind2Dialogue introduces a psychology‑guided simulator that generates user mental states and uses privileged distillation to train LLM assistants on Oracle responses that know those states. The resulting models improve personalization and theory‑of‑mind metrics by up to 41 percentage points versus standard instruction‑tuned baselines.

    Hugging Face Daily Papersarxiv.org2 minpaper