proomt

Search

Search posts, papers, and topics

All posts

VercelElisabeth Rülke1 min readrelease notesintro

Run Terminal-Bench and other Harbor evals on Vercel Sandbox

Read on vercel.com

Summary

Vercel now supports running Harbor benchmarks, like Terminal‑Bench, inside its Sandbox using Firecracker microVMs. The integration with AI Gateway lets you benchmark multiple models by swapping the `--model` flag, scaling concurrency with `--n-concurrent`.

  • Harbor benchmarks can be executed on Vercel Sandbox via `--env vercel`, each trial runs in an isolated Firecracker microVM.
  • Network policies are enforced at the sandbox firewall; credential injection adds secrets only to outbound requests, never inside the VM.
  • Vercel AI Gateway lets a single `AI_GATEWAY_API_KEY` access many models; switch models with the `--model` flag.
  • Install with `uv tool install 'harbor[vercel]'` and set `VERCEL_TOKEN` and `AI_GATEWAY_API_KEY` before running.

AI engineers and researchers can run large‑scale model benchmarks without provisioning their own infrastructure.

4/10
devopscloudaibenchmarking

Related reading

  1. Vercel Sandbox now supports Devin Outposts

    Vercel Sandbox adds support for Devin Outposts, letting each AI agent session run in an isolated microVM with snapshot state and firewall controls. Teams can deploy via a provided quickstart and get automatic credential handling.

    Vercelvercel.com1 minrelease

  2. Full Sandbox egress firewall now available on Hobby plan

    Vercel added full Sandbox egress firewall capabilities to the Hobby plan, letting you define allow/deny network policies and attach secrets to outbound requests. You can configure policies via code or the Sandbox CLI, reducing data exfiltration risk for low‑cost projects.

    Vercelvercel.com1 minrelease

  3. How Factory scaled its cloud backend to one billion monthly requests on Vercel

    Factory runs its entire SaaS backend on a single Next.js app deployed to Vercel, handling about a billion API requests per day with 350 ms p95 latency. They achieve this with Vercel’s Fluid compute to avoid cold starts, built‑in observability and WAF for security, and internal automation (“Droids”) that let non‑engineers deploy tools without extra infrastructure.

    Vercelvercel.com4 min

  4. Next.js 16.3 support on Vercel

    Vercel added full support for Next.js 16.3, bringing leaner prefetching, immutable static assets, and faster route metadata handling. Early adopters see up to 45% fewer prefetches, 24% less static traffic, and roughly 2× faster routing at scale.

    Vercelvercel.com4 min

  6. SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops

    SiliconBench benchmarks nine Apple‑Silicon LLM serving engines on unified‑memory desktops, measuring speed, memory usage, and output fidelity across Qwen3, Qwen3.5, and Gemma‑4 models. It finds vllm‑metal doubles throughput at modest concurrency, memory budgets often fail to preserve headroom, and only three stacks satisfy all fidelity and model‑coverage requirements, with tensor‑parallel scaling…

    Hugging Face Daily Papersarxiv.org1 minpaper