Hugging Face Daily PapersChuxuan Hu, Yeye He, Penny Zhou, Wee Hyong Tok, Daniel Kang, Surajit Chaudhuri2 min readpaperadvanced
BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence
Summary
The paper introduces BI‑Bench, a new benchmark of real‑world BI questions derived from public dashboards, and BI‑Agent, a tool‑augmented LLM system that breaks BI workflows into search, join, and transform subtasks. Baseline LLMs hit <50 % accuracy on BI‑Bench. By orchestrating specialized data‑management tools and post‑training the model with supervised fine‑tuning and reinforcement learning on…
- BI‑Bench provides the first systematic evaluation of LLMs on end‑to‑end business‑intelligence tasks, using real questions and ground‑truth answers from actual dashboards.
- Even state‑of‑the‑art LLMs struggle (<50 % accuracy) on this benchmark, exposing gaps in data‑aware reasoning.
- BI‑Agent decomposes the BI pipeline into discrete operations (table search, join inference, data transformation) and calls specialized data‑management modules, enabling the LLM to focus on high‑level orchestration.
- A post‑training pipeline synthesizes realistic BI task trajectories from the benchmark, allowing supervised fine‑tuning (SFT) and reinforcement learning (RL) to teach the model effective tool use.
Enterprise BI tools (Power BI, Tableau) still require manual data wrangling, a bottleneck for analysts. Demonstrating that LLMs can be turned into competent BI assistants—when equipped with the right toolchain and fine‑tuned on realistic workflows—opens a path to more autonomous analytics platforms…
8/10
