proomt

Search

Search posts, papers, and topics

data engineering

RSS
  1. 2

    Show HN: Scry, programmable internet search w/ congestion pricing

    Scry is a read‑only, SQL‑like search service that lets LLM agents query billions of public‑web records (e.g. Reddit, Hacker News, academic papers) via a single MCP or HTTP endpoint. It enforces per‑query limits (deadline, memory, row cap), offers schema discovery, explain, rerank, and compile APIs, and reports freshness per source (e.g. Hacker News <15 min). In a DeepSearchQA benchmark Scry‑enabl…

    Hacker News front pagescry.io28 minreleaseHN6025
  2. 3

    WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsing

    WeVisDoc introduces a two‑stage data‑centric pipeline for end‑to‑end document parsing. Stage I expands coverage using heterogeneous data and structure‑preserving degradations. Stage II probes the Stage I model with a held‑out set, clusters residual errors, and directs targeted data creation and token‑budget reallocation. The 4‑billion‑parameter model reaches 95.38 Overall on OmniDocBench v1.6 and…

    Hugging Face Daily Papersarxiv.org1 minpaper
  3. 6

    Glyph: A Multi-Strategy Agentic System for Column Description and Sensitivity-Ontology Tagging of Enterprise Data Catalogs

    Glyph is a production system that uses coordinated LLM agents and a fine‑tuned MiniLM encoder to automatically generate column descriptions and assign ontology tags in enterprise data catalogs. It combines code‑grounded retrieval, regex, and contrastive vector search, achieving NDCG@10 0.92 and MAP@100 0.90, and provides auditable provenance for each tag.

    Apple Machine Learning Researchapple.com1 minpaper
  4. 7

    MilleMiglia: A realistic instance generator for middle-mile logistics

    MilleMiglia is an open‑source C++ generator that creates realistic, privacy‑preserving middle‑mile logistics instances (space‑time graphs with fixed schedules, throughput limits, and synchronization constraints). It uses data‑driven spatial, demand, and rotation distributions, serializes with protobuf, and ships small files for small‑toy to continent‑scale problems, enabling both exact/heuristic…

    Google Researchresearch.google7 min
  5. 8

    How Data 360 Builds Trusted Context: The Enduring Layer for Enterprise AI

    Salesforce’s Data 360 provides a shared runtime that assembles the minimal, authorized slice of enterprise data (“Trusted Context”) for each AI‑agent turn. A six‑stage Agent Context Engine (Resolve, Plan, Reconcile, Govern, Compile, Learn) pulls data from structured, unstructured, and streaming sources across Salesforce, Snowflake, Databricks, etc., applies fine‑grained policy, and returns a toke…

    Salesforce Engineeringsalesforce.com11 min
  6. 9

    The DevFest Community Workshop Experience: Building Real Agents Together

    Google’s DevFest Community Workshop introduced a “Workbench” format that emphasizes architectural mental models over copy‑paste code, guiding engineers to build long‑running, self‑evolving multi‑agent systems with the Agent Development Kit and Gemini Enterprise platforms. Attendees learned state‑separation, workflow pausing, and self‑patching pipelines, and the series will continue in five more c…

    Google Cloud Bloggoogle.com2 min
  7. 10

    How Databricks’ marketers use data 3x more with Genie, an AI analytics assistant

    Databricks built Marge, a Genie‑powered conversational analytics assistant on a governed Marketing Lakehouse. By starting with a single high‑value use case (email campaign performance), documenting data, encoding verified answers, teaching business terminology, and embedding the tool in existing ticket workflows, they achieved 85% adoption, 3× higher data usage in decisions, 50% QoQ usage growth,…

    Databricksdatabricks.com10 min
  8. 12

    How energy teams turn theft detection into governed action with Genie and AI business processes

    Databricks shows how to turn energy‑theft ML scores into a governed, end‑to‑end workflow using a Databricks App, Lakebase for live case state, Unity Catalog for data governance, and Genie One for natural‑language executive reporting. The pattern lets utilities act on alerts faster while staying compliant, and can be reused for other fraud‑type use cases.

    Databricksdatabricks.com6 min
  9. 13

    RADAR: Catch gray failures with anomaly detection

    Databricks built RADAR, a four‑stage, metric‑agnostic pipeline that uses streaming anomaly detection (SPOT) to surface gray failures in minutes with >90% precision. The blog shows how to recreate the system on Databricks for any metric, from billing to model drift.

    Databricksdatabricks.com7 min
  10. 14

    datasette 1.0a40

    Datasette 1.0a40 adds a background‑task API (datasette.add_background_task()), switches the internal HTTP client to httpx2, and ships a batch of bug fixes aimed at stabilising the upcoming 1.0 release. The security fix mirrors that of 0.65.5.

    Simon Willisonsimonwillison.net1 minrelease
  11. 15

    "Regex for Rows": Simplifying Pattern Detection in SQL with MATCH_RECOGNIZE

    Databricks announces MATCH_RECOGNIZE, a preview‑only SQL clause that lets you write regex‑style patterns over ordered rows. The post walks through four industry‑level use cases (security login‑failure sequences, V‑shaped stock trends, e‑commerce cart abandonment, IoT sensor‑driven failure prediction) and claims the operator replaces complex CTEs, window functions, and self‑joins. No actual syntax…

    Databricksdatabricks.com5 min
  12. 16

    ClickHouse welcomes RunReveal

    ClickHouse announced it has acquired RunReveal, a security data platform built on ClickHouse, to deepen its expertise in security workloads. The deal promises better guidance for security teams while keeping RunReveal's product unchanged for existing customers.

    ClickHouseclickhouse.com2 minHN7
  13. 17

    datasette 0.65.5

    Datasette 0.65.5 releases with a security fix: a trailing newline in a table name could bypass permissions and expose private rows (GHSA‑h547‑rmjf‑5m2m).

    Simon Willisonsimonwillison.net1 minrelease
  14. 18

    System Design Interviews for Data Roles: What to Actually Practice

    The piece shows that data‑role system design interviews evaluate how you turn vague requirements into a defensible architecture, not which tools you name, and it gives a concrete prep framework: clarify scope, quantify load, pick batch vs streaming with trade‑offs, and address failure handling. Candidates should rehearse explaining these decisions aloud with numbers rather than just drawing diagr…

    SitePointsitepoint.com6 min
  15. 19

    Writing the Docs: 2026 Edition

    The post revisits core documentation principles—separating what from how, grounding reference docs in source truth, and continuously improving—and examines how large language models fit into this workflow. It argues that diataxis still guides effective docs and that LLM‑generated drafts must be anchored to machine‑readable ground truth to avoid hallucination.

    Mozilla Automation Teamwrla.ch15 min
  16. 21

    Glean Dictionary + DuckDB

    A short post describing how the Mozilla Glean Dictionary JSON can be exported into a DuckDB file via a simple ETL script, enabling SQL queries and notebook‑driven analysis of metric types and growth over time.

    Mozilla Automation Teamwrla.ch2 min
  17. 22

    Qubika Strengthens Its Position in Financial Services with Databricks Specialization

    Qubika announced its Databricks Financial Services Specialization, highlighting past engagements modernizing data lakes, migrations, and AI foundations for banks, fintechs, and lenders. The post lists partner credentials, analyst recognitions, and generic benefits of a unified data‑AI platform but provides no technical details, code, or measurable outcomes.

    Moove-itqubika.com4 min