proomt

Search

Search posts, papers, and topics

All posts

InfoQSiddharth Kodwani, Swaroop Chitlur33 min readtalkintermediate

Presentation: Building GenAI Platform at DoorDash

Read on infoq.com

Summary

DoorDash built a unified GenAI platform with an LLM Gateway that abstracts multiple model providers, adds cost attribution, and offers centralized observability. The platform now serves over 5,000 internal users, including many non‑engineers, with rapid onboarding and reliable fallbacks.

  • Implemented a single LLM Gateway API/SDK to route requests across OpenAI, Anthropic, Gemini, and open‑source models with automatic fallback handling.
  • Added workspace‑based cost attribution and usage tracking, enabling finance visibility for 5k+ users.
  • Shifted focus to API‑first, product‑engineer customers, resulting in 45 new daily onboardings and 40% non‑engineer adoption.
  • Centralized logging and observability through the gateway UI, reducing per‑team instrumentation effort.

Platform and ML‑ops engineers can learn practical strategies for scaling multi‑model GenAI services with cost control, reliability, and observability.

6/10
llmapi designobservabilityplatformcost managementgenai

Related reading

  1. DoorDash Uses Multi Agent LLMs to Clean up 60,000 Feature Flags

    DoorDash built a two‑phase, multi‑agent LLM system (Claude Sonnet orchestrator + Claude Opus cleanup agents) that automatically removes stale feature flags from 623 repos. In a 50‑flag pilot it generated usable PRs for 45 flags at ~14 min and $4.79 each, with 100 % success on simple flags and no regressions.

    InfoQinfoq.com2 min

  3. LLMs belong in your backend

    Neon AI Gateway integrates LLM calls directly into the Neon backend, offering a unified API for various models (frontier and open-weight) and consolidating billing. It extends Neon's branching capabilities to LLM endpoints, allowing each development branch to have its own isolated AI Gateway host and credentials.

    Neonneon.com7 min

  4. The Genie One MCP is now Generally Available

    Databricks announced the general availability of the Genie One Model Context Protocol (MCP) server, a managed service that exposes Genie One’s unified business context (via Genie Ontology) to any AI agent through Unity Gateway, enabling consistent, governed access to structured and unstructured data. The post lists high‑level use cases (slide generation, customer outreach, developer tooling) but…

    Databricksdatabricks.com4 minrelease

  5. GPT-Live 1 now available on AI Gateway

    Vercel AI Gateway now offers OpenAI’s GPT‑Live 1, a full‑duplex voice model that can listen and speak simultaneously. The SDK lets you start a voice session and optionally delegate work to any text model, with code examples for both patterns.

    Vercelvercel.com1 minrelease

  6. Article: Beyond Relevance: A Governance-First Architecture for Enterprise Personalization

    The article proposes a governance‑first architecture for enterprise personalization, where policy‑driven steps (memory, journey graph, AI routing, scoring, trust checks, outcome simulation) shape the recommendation before it is returned. A reference FastAPI implementation demonstrates the pattern with external YAML policies and optional LLM assistance.

    InfoQinfoq.com19 min