proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersFelipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange1 min readpaperadvanced

Towards a Deterministic Math Solver for Clinical Language Models

Read on arxiv.org

Summary

The paper proposes a Program‑Solve interface where LLMs emit Python code that a sandboxed executor runs, making arithmetic deterministic for clinical calculators. Experiments show the approach boosts a 32B model’s accuracy on a verified benchmark, but coverage and upstream extraction remain limiting factors.

  • The Program‑Solve interface has the LLM generate case‑specific Python code that runs in a sandbox, turning arithmetic into deterministic execution rather than relying on the model's own calculations.
  • On the MedCalc‑Bench Verified benchmark the 32B model improves from 83.47% to 90.53% accuracy (+7.05 points, statistically significant), while the 7B model shows a non‑significant 3.29‑point gain.
  • A hand‑written 22‑calculator library is perfectly accurate on its 440 supported cases but only covers 40% of the benchmark, illustrating the trade‑off between coverage and correctness.
  • Even with perfect formula provision, errors in variable extraction or outdated clinical coefficients still limit overall reliability; the executor cannot compensate for upstream mistakes.

Clinical AI engineers and LLM developers should care because deterministic numeric computation is essential for safe medical decision support, and this work shows a practical way to improve reliability without hand‑coding every calculator.

7/10

Related reading

  1. Register Tokens for Bounded-State Reasoning in Diffusion Language Models

    Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Learning to solve hard problems in RL for LLMs by never giving up

    The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…

    Hacker News front pagegithub.io11 minHN1179

  4. How we AI-updated 1,138 Error-code Docs Pages

    Twilio built a two‑phase pipeline that first scores each error‑code doc with a deterministic script, then feeds only low‑scoring pages to an LLM (gpt‑5.4) using a strict prompt. The model’s diff is applied via git, linted, and opened as a PR for a reviewer. In 12 weeks they updated 1,138 pages (≈34.6 s/page) with a human reviewing every change, cutting manual effort from an estimated 18 k hours a…

    Twiliotwilio.com11 min