proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersXiaoya Wang, Yutong Xu, Junjie Wang1 min readpaperintermediate

E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning

Read on arxiv.org

Summary

E2A‑Bench is a 969‑query benchmark for financial chart reasoning that evaluates vision‑language models across a full evidence‑to‑action chain using four metrics (UCR, RCI, ECI, NDR). Experiments on 20 VLMs expose hidden failures: low‑UCR models have only 6.4 % directional coverage, oracle‑aided verification cuts unsupported claims but can kill coverage, and fine‑tuning inflates BUY:SELL ratios by…

  • Benchmark design: 969 queries derived from 323 HS300 stocks, three input modalities, deterministic OHLCV evidence anchors.
  • Metrics: UCR (grounding), RCI (reasoning‑action consistency), ECI (evidence‑confidence calibration), NDR (directional coverage‑aware reliability).
  • Findings: Models with high grounding can still rank low on NDR due to poor directional coverage; oracle verification improves claim support but may reduce coverage; financial fine‑tuning skews BUY/SELL predictions drama…
  • Implication: Evaluation of financial VLMs must trace the entire evidence‑to‑action pipeline rather than rely on scalar hallucination scores.

Financial decision‑making systems need traceable, calibrated recommendations. By exposing gaps between grounding, confidence, and actionable output, E2A‑Bench provides a more realistic safety net for deploying VLMs in trading or advisory contexts.

8/10
llmevaluationbenchmarkvision language modelsfinance

Related reading

  1. Modernizing the Trade Lifecycle With Governed Data and AI

    Databricks argues that modernizing the trade lifecycle now hinges on building a governed, real‑time data foundation that spans research, trading, risk, ops and compliance, rather than isolated AI pilots. Starting with a few high‑value questions—execution cost, shock risk, exception rates—and using Unity Catalog and Agent Bricks lets firms achieve measurable speed and auditability gains before sca…

    Databricksdatabricks.com5 min

  2. TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

    The paper introduces TeleAntiFraud 2.0, a monthly refreshed, frozen audio benchmark of 900 Chinese telecom calls (600 fraud, 300 near‑domain non‑fraud) built with a Mixed‑Tree generation pipeline. Experiments show models that score perfectly on unrelated negatives fall to ~0.66 Macro‑F1 on near‑domain cases, exposing shortcut learning and prediction collapse.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. When2Think: Learning Difficulty-Aware Length Control for Efficient Hybrid Reasoning Models

    When2Think introduces a post‑training framework that lets a large reasoning model decide per‑instance how much reasoning depth to allocate, using difficulty‑aware reward shaping (IDAC) and verifier rewards. It cuts token usage by ~28% while boosting Pass@3 by 10% on AIME24 and reaches 40% Pass@3 on AIME25, outperforming compression and routing baselines.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Laya the open source version of Jev

    Laya is an open‑source, bidirectional‑encoder model family for ultra‑fast, calibrated decision‑making (choice, score, boolean) over structured schemas. It runs 6‑8× faster than the closed‑source Jev, supports 100+ languages via three checkpoints, and includes a lightweight router that selects the appropriate checkpoint before inference. Benchmarks show higher accuracy, far better calibration (ECE…

    Hacker News front pageconvaiinnovations.com8 minreleaseHN1326313lobste.rs2

  5. ‘Now We Can Know Everything and Do Anything,’ Jensen Huang Says at Dreamforce

    Nvidia’s Jensen Huang announced Salesforce’s Koa, a CRM‑reasoning LLM built by fine‑tuning Nvidia Nemotron 3 Super on a synthetic, three‑decade‑spanning dataset. Koa uses supervised fine‑tuning plus RL (NeMo RL, Gym, AutoModel), covers 14+ industries, and claims 3× fewer errors on Salesforce’s CRM‑Bench versus leading models. It’s already in internal Slack agents and slated for limited customer p…

    Nvidianvidia.com3 min

  6. Article: Architecting Secure and Scalable Facial Verification Systems

    A real‑world post‑mortem of a high‑volume face verification service that moved from a naïve synchronous API to an async, layered pipeline (edge validation, preprocessing, decoupled detection/verification, decision engine) to achieve 8.5k rpm, p99 < 1.8 s, 30 % cost savings, and strict privacy controls.

    InfoQinfoq.com15 min