proomt

Search

Search posts, papers, and topics

All posts

Hacker News front page2625882138434761 min readtutorialintermediate

Laya (OS Jev) on Mac M4 CoreML Offline (45 decisions per second)

Read on github.comHN 166·32 comments

Summary

A short walkthrough showing how to install and run the Laya‑CoreML demo (a multilingual language model) on a Mac M4 using the uv package manager, with a video of the UI, memory‑usage snapshot, and a sample Cloudflare JEv API call that returns a single urgency score.

  • Installation is a three‑step uv workflow: `uv init`, `uv add 'laya-coreml[demo]'`, and `uv run laya-coreml-snake` after downloading the model from Hugging Face.
  • The demo runs at ~45 decisions per second on an M4, using ~560 MiB RAM (peak ~780 MiB).
  • A simple HTTP POST to the public `laya.inference.zaitlabs.com` endpoint returns a JSON with a probability‑like score (e.g., 0.7894 for urgency).

Shows that on‑device CoreML can serve a multilingual LLM with sub‑second latency on consumer‑grade Apple silicon, useful for privacy‑preserving or edge AI deployments.

3/10
llmpythonperformanceml inframacoscoreml

Related reading

  1. Laya the open source version of Jev

    Laya is an open‑source, bidirectional‑encoder model family for ultra‑fast, calibrated decision‑making (choice, score, boolean) over structured schemas. It runs 6‑8× faster than the closed‑source Jev, supports 100+ languages via three checkpoints, and includes a lightweight router that selects the appropriate checkpoint before inference. Benchmarks show higher accuracy, far better calibration (ECE…

    Hacker News front pageconvaiinnovations.com8 minreleaseHN1326313lobste.rs2

  2. OpenJev

    OpenJev is a browser‑only demo that lets you load small LLM checkpoints (e.g., MiniCPM‑5 2B, Qwen3 0.6B) onto your GPU and compare two inference paths: reading raw logits for a set of options versus prompting the model to emit a JSON with option probabilities token‑by‑token. The page reports model sizes, download times, balanced accuracy on a few benchmarks, and wall‑clock timings measured with `…

    Hacker News front pageopenjev.com2 minreleaseHN709288

  3. M5 Ultra Mac Studio Review

    The M5 Ultra Mac Studio (256 GB RAM) uses a quad‑die M5 Max architecture with an 80‑core GPU and 1.2 TB/s memory bandwidth, delivering ~70 % faster prompt‑to‑first‑token and token‑generation rates than the M3 Ultra. In the author’s tests Qwen3.8‑Flash‑Next hits 100 tokens/s on short prompts and 60‑85 tokens/s with 64‑256 KB context, making local AI agents (Open Minis, Hermes, Codex) feel snappy a…

    Hacker News front pagemacstories.net39 minHN191174

  5. Introducing System One Models and Jev

    TypeSafe AI announced its first “System One” model, Jev, a non‑text‑generating LLM that outputs type‑safe structured decisions with calibrated probabilities. It claims 40‑200× lower latency (70‑500 ms) and 100‑500× lower cost versus frontier LLMs, no hallucinations, and parallel sampling. The post includes a side‑by‑side demo, a custom “workflow” benchmark comparing Jev to GPT‑5.6/6 and other mod…

    Hacker News front pagetypesafe.ai9 minHN1824480lobste.rs26

  6. Quiz: How to Get Started With Ollama

    This is a 10‑question quiz that checks your grasp of installing Ollama, pulling models, choosing between chat and generate APIs, and managing multi‑turn conversations in Python. It reinforces the basics of running LLMs on your own hardware for privacy and offline use.

    Real Pythonrealpython.com1 min