proomt

Search

Search posts, papers, and topics

All posts

NeonCarlota Soto7 min readintermediate

LLMs belong in your backend

Read on neon.com

Summary

Neon AI Gateway integrates LLM calls directly into the Neon backend, offering a unified API for various models (frontier and open-weight) and consolidating billing. It extends Neon's branching capabilities to LLM endpoints, allowing each development branch to have its own isolated AI Gateway host and credentials.

  • Neon AI Gateway unifies access to multiple LLM providers (OpenAI, Anthropic, open-weight) via a single API key and endpoint.
  • LLM usage is consolidated onto a single Neon bill, with no markup over provider pricing.
  • AI Gateway endpoints branch with the rest of the Neon stack, providing isolated LLM environments for preview, feature, and CI branches.
  • It leverages Databricks Foundation Model APIs for scalable and performant LLM serving, including features like prompt caching.

Developers building LLM-powered applications can simplify their infrastructure, credential management, and billing by integrating model calls directly into their backend services.

6/10
devopsllmapi designdatabasescloud nativeai infra

Related reading

  1. Open-weight models are fast on Neon AI Gateway. Here's why

    Neon AI Gateway provides fast inference for open-weight LLMs by utilizing Databricks' optimized serving stack. This stack incorporates techniques like prompt caching, continuous batching with KV-cache paging, and a TensorRT-LLM backend, leading to significant improvements in throughput and latency.

    Neonneon.com6 min

  2. GLM 5.3 FlashX now available on AI Gateway

    Vercel AI Gateway now offers the GLM‑5.3‑FlashX model, a fast (~200 tps) multimodal coding LLM. The post includes a TypeScript streaming example, notes the model’s fit for coding agents and interactive tools, and lists AI Gateway features (unified API, usage tracking, retries/failover, custom reporting, key budgets, routing rules) with no platform fee.

    Vercelvercel.com1 minrelease

  3. Query Neon backend logs

    Neon has introduced new capabilities to query backend logs for Neon Functions and Object Storage outside of the console. This includes a new CLI, API, SDK, and direct Loki endpoint access, allowing unified log inspection across different backend services on a branch.

    Neonneon.com3 min

  4. Neon skills landed in the Vercel CLI

    Vercel CLI now integrates Neon, allowing users to provision a Neon database and install its agent skills with a single command. This streamlines database setup for Vercel projects, including automatic isolated branches for preview deployments.

    Neonneon.com4 minrelease

  5. Gemini 3.8 Live models now available on AI Gateway

    Vercel AI Gateway now offers Google’s Gemini 3.8 Live and Live Extended Thinking models through its realtime WebSocket API, enabling real‑time spoken interactions with audio output and parallel reasoning. The post includes a short TypeScript example for token acquisition, WebSocket setup, and handling of realtime events.

    Vercelvercel.com1 minrelease

  6. Neon is now available in Grok Bot

    Neon released a Grok Bot plugin that lets the AI assistant manage Neon PostgreSQL databases. After a one‑time auth, the bot can create projects, spin up isolated branches, run SQL, list databases, and set up usage‑summary routines directly from chat.

    Neonneon.com1 min