Open Source AI Gateway

What is it

An Open Source AI Gateway is a self-hostable proxy layer that sits between your application and multiple AI model providers—OpenAI, Anthropic, Google Gemini, local models via Ollama, and others. It centralizes API key management, request routing, load balancing, rate limiting, caching, observability, and cost tracking into a single configurable service. Think of it as Kong or NGINX, but purpose-built for LLM traffic.

The technical essence: instead of hardcoding provider SDKs into your app, you point your app at one gateway endpoint. The gateway handles failover, retries, prompt logging, and token accounting. If OpenAI goes down or raises prices, you switch providers by changing a config file—not by rewriting code.

The business significance is twofold. First, it eliminates vendor lock-in, which matters more as enterprises realize their AI spend is concentrated in one billing relationship. Second, it creates a natural choke point for observing and controlling AI costs—the fastest-growing line item in many software budgets. For indie developers, this is a classic "picks and shovels" play: you don't need to build a model or an app; you build the plumbing that everyone using models needs.

Why now

Three forces converge to make this the right moment. First, the LLM provider market has fragmented dramatically. In 2023, OpenAI was the only serious option. In 2026, you have Anthropic's Claude, Google's Gemini, Meta's Llama via multiple hosts, Mistral, Cohere, plus a wave of cheaper open-weight models on Together, Fireworks, and Groq. Each has different pricing, latency, and quality profiles. Developers need an abstraction layer to manage this complexity without going insane.

Second, the cost crisis is real. Companies are hitting $50K–$500K monthly AI bills and discovering that naive integration code produces massive waste—duplicate calls, no caching, no model tiering. A gateway that routes simple queries to a cheap model and complex reasoning to a premium model can cut costs 40–70%. That ROI math sells itself.

Third, the regulatory and compliance pressure has shifted. GDPR, HIPAA, and the EU AI Act create real legal liability when your data goes to a US-based provider. Self-hosted gateways with local model support let European companies keep data in-region. This wasn't a board-level concern two years ago; now it is.

The timing window is real but narrow. The market is nascent, but Vercel, Cloudflare, and Kong are already circling. You have 6–12 months to establish a differentiated position before consolidation.

Market Evidence

The data is thin: 2 sources, 2 mentions, 100% growth rate, nascent stage. I will not sugarcoat this—those numbers are not proof of demand. They are proof that someone, somewhere, started talking about this term recently. The opportunity score of 0/100 and demand score of 0/100 reflect that no search volume, no community traction, and no revenue signals exist yet.

However, the trend score of 64/100 and the 100% growth rate (from 1 to 2 mentions) indicate early signal, not noise. The pattern matches how developer tool categories typically emerge: a few open-source projects appear on GitHub and Product Hunt, gain stars, then commercial players enter 6–12 months later.

Real demand evidence comes from adjacent categories. LiteLLM, an open-source LLM proxy, has over 15,000 GitHub stars and is used in production at hundreds of companies. Portkey has raised venture funding for its AI gateway product. Helicone, focused on observability, passed $1M ARR. These prove developers will adopt gateway tooling. The specific term "Open Source AI Gateway" is nascent, but the underlying need is validated.

My position: this is early but real. The absence of search volume means low competition for SEO and positioning. The presence of adjacent successes means the category will grow. Act now, not later.

Who's Behind It

The named projects—OmniRoute and Experiential Labs—are small, likely solo or two-person open-source efforts. Neither has meaningful brand recognition. This is typical for the nascent stage: hobbyists building tools for their own pain, publishing them, and seeing modest adoption.

The real whales are not the open-source projects; they are the commercial players circling the same problem. Portkey (funded, ~$5M raised) builds a hosted AI gateway with enterprise features. LiteLLM is the open-source community darling, backed by BerriAI, and has become the default choice for many startups. Kong, the API gateway incumbent, announced AI gateway features in 2024. Cloudflare has AI Gateway as part of its developer platform. Vercel's AI SDK includes gateway-like routing capabilities.

The competitive dynamic: the open-source projects (OmniRoute, Experiential Labs, LiteLLM) compete on being free and self-hostable. The hosted players compete on convenience and enterprise features. The platform giants (Cloudflare, Vercel) bundle gateway functionality into their existing ecosystems, betting on lock-in rather than standalone value.

Your opening: the open-source projects lack polish, documentation, and enterprise features. The hosted players require sending traffic through third-party infrastructure—defeating the self-hosted compliance purpose. There is room for a well-designed, truly self-hosted gateway with excellent DX.

TAM & Market Size

The buyer universe splits into three segments. First, indie developers and small startups (1–50 employees) building AI features. They spend $500–$10K monthly on LLM APIs. They want to avoid lock-in and control costs but have limited DevOps capacity. They will pay $0–$100 monthly for tooling. Estimate: 500,000 such developers globally in 2026.

Second, mid-market companies (50–500 employees) with AI initiatives. They spend $10K–$100K monthly on LLM APIs. They need governance, audit trails, and compliance. They will pay $500–$2,000 monthly for a gateway with support. Estimate: 20,000 such companies.

Third, enterprises (500+ employees). They spend $100K+ monthly on AI. They need SSO integration, on-prem deployment, and SLAs. They will pay $2,000–$10,000 monthly. Estimate: 3,000 such companies.

Total addressable market: 523,000 potential buyers. At conservative average revenue of $30/month across the full base, that is $188M annually. Realistic serviceable obtainable market for a focused indie founder in year two: 200–500 paying customers at $50–$500 monthly, yielding $120K–$3M ARR.

Will they pay? Yes—the cost savings a gateway delivers (30–70% of LLM spend) dwarf the subscription cost. The ROI conversation is easy. The challenge is not convincing buyers of value; it is reaching them before they default to LiteLLM's free tier.

Competitive Landscape

LiteLLM is your primary threat. It is open-source, free, supports 100+ providers, and has strong community momentum. Its weaknesses: the codebase is sprawling, the documentation is mediocre, and it lacks polished observability and analytics. It is a developer tool, not a product.

Portkey is the commercial hosted option. It has decent features but requires sending traffic through Portkey's cloud, which defeats self-hosting for compliance-sensitive buyers. Pricing starts around $49/month.

Cloudflare AI Gateway is bundled with Cloudflare's developer platform. It is cheap (usage-based) and well-integrated, but it is Cloudflare-specific—if you are not on Cloudflare, it adds complexity.

Vercel AI SDK is excellent for Next.js developers but assumes you are building on Vercel. It is a framework, not an infrastructure layer.

Kong's AI gateway is enterprise-focused, expensive, and overkill for small teams.

The gap: a self-hosted, TypeScript-native gateway with beautiful observability, cost analytics, and a clean UI—positioned between LiteLLM's raw utility and Portkey's hosted lock-in. You cannot out-feature LiteLLM's community momentum. You can out-polish it. You cannot out-price Cloudflare. You can out-focus them by being the best standalone solution.

Time before Big Tech consolidates: 12–18 months. Cloudflare or Vercel could absorb the category. Your window is real but finite.

Business Model

Recommended model: open-core with a hosted SaaS tier. The core gateway (routing, caching, basic analytics) is open-source under MIT or Apache 2.0. This drives adoption and community contributions. The commercial tier is a hosted control plane—the UI dashboard, advanced analytics, team management, SSO, and multi-project support—sold as SaaS at $49–$199 per month.

Why this model fits: developers evaluating gateways will not pay before trying. Open-source removes friction. But once a team runs the gateway in production, the pain of managing multiple gateway instances across projects creates willingness to pay for centralized management.

Pricing structure:

• Free (open-source core): unlimited gateway instances, basic routing, community support
• Pro at $49/month: hosted dashboard, 3 projects, cost analytics, 30-day log retention
• Team at $199/month: 15 projects, SSO, audit logs, priority support, 90-day retention
• Enterprise at $999+/month: on-prem control plane, custom SLAs, dedicated support

Twelve-month revenue forecast:

• Conservative: 150 Pro + 30 Team = $7,350 + $5,970 monthly = $159K ARR
• Base: 400 Pro + 100 Team = $19,600 + $19,900 monthly = $474K ARR
• Optimistic: 800 Pro + 250 Team + 15 Enterprise = $39,200 + $49,750 + $14,985 monthly = $1.25M ARR

CAC estimate: $150–$300 per customer through content marketing and GitHub presence. Payback period: 3–6 months at Pro pricing. This is a high-margin business—marginal cost per customer is near zero.

MVP Blueprint

Estimated dev days: 0 in the data, but realistically 5–7 days for a solo developer with TypeScript experience. Do not build more than this.

Core features ONLY:

1. OpenAI-compatible API endpoint that proxies requests to OpenAI, Anthropic, and Gemini
2. Config-based routing rules (e.g., "route all requests with max_tokens < 500 to Gemini Flash")
3. Simple round-robin or priority-based fallback between providers
4. Response caching with configurable TTL
5. Token counting and cost logging per request
6. A minimal REST API for viewing logs and usage stats

Cut entirely: UI dashboard, SSO, team management, advanced analytics, prompt templating, model evaluation, A/B testing.

Tech stack:

• TypeScript + Node.js (Fastify or Hono for the HTTP layer)
• SQLite for logs (replace with Postgres later)
• Redis only if you need distributed caching (skip for MVP—use in-memory TTL cache)
• Deploy as a single Docker container

Fastest path to launch:

1. Day 1–2: Build the proxy core. The OpenAI SDK is compatible with Anthropic and Gemini via their OpenAI-compatible endpoints, so you can proxy with minimal transformation code.
2. Day 3–4: Add routing logic and caching.
3. Day 5: Add cost tracking. Every provider returns usage data in the response—log it.
4. Day 6–7: Write documentation, create a demo video, publish to GitHub and Product Hunt.

Do not build the SaaS dashboard in the MVP. Launch the open-source core first, gauge adoption, then build the commercial tier.

Commercial Opportunities

Direction 1: Cost Optimization Add-On A paid module that analyzes your LLM usage patterns and automatically suggests routing rules to cut spend. Target persona: engineering leads at startups spending $5K+ monthly on AI APIs. Revenue range: $100–$500 monthly per customer. This beats alternatives because cost savings are measurable—you can show the customer their first month's savings exceeded the subscription price.

Direction 2: Compliance-Focused Enterprise Edition An on-prem deployment package with audit logging, data residency guarantees, and SSO integration. Target persona: CTOs at European healthcare or finance companies bound by GDPR or HIPAA. Revenue range: $1,000–$5,000 monthly. This beats generic hosted gateways because you physically cannot send data through a third-party cloud if compliance forbids it. Self-hosted is the only option, and you are the polished self-hosted player.

Direction 3: Usage Analytics and Chargeback A dashboard that shows per-project, per-team, per-feature AI costs, enabling internal chargeback. Target persona: platform engineers at companies with 10+ developers building AI features. Revenue range: $200–$800 monthly. This beats building in-house because every company that scales AI usage eventually needs this and would rather buy than build.

Product Ideas

🥇 Gateway with Automatic Model Routing Value prop: "Your requests automatically go to the cheapest model that meets your quality threshold." Target user: startups with growing AI bills who cannot manually tune routing. Why now: model quality differences between cheap and expensive models are narrowing monthly, making automatic routing viable. You can use a small classifier model to predict whether a request needs premium reasoning.

🥈 Self-Hosted Gateway with Built-in Fine-Tuning Cache Value prop: "Cache responses AND fine-tune on repeated patterns to reduce future costs." Target user: companies running high-volume, repetitive AI tasks like support classification or content extraction. Why now: fine-tuning costs have dropped, and the economics of caching similar prompts plus fine-tuning on the patterns is newly viable.

🥉 Gateway with Multi-Cloud Failover Dashboard Value prop: "One dashboard showing all provider status, latency, and cost, with automatic failover." Target user: platform teams running production AI features with uptime requirements. Why now: providers have had multiple high-profile outages in 2025–2026, and the cost of downtime is now board-level visible.

SEO Opportunity

Search volume is near zero today—SEO difficulty is 0/100, which is an opening, not a barrier. You can own this keyword space before it grows.

Target long-tail keywords:

• "open source ai gateway" (0–100 monthly searches, low competition)
• "self hosted llm proxy" (100–300 monthly searches)
• "ai gateway vs direct api" (50–150 monthly searches)
• "reduce openai api costs self hosted" (100–400 monthly searches)
• "llm routing open source" (50–200 monthly searches)

Content strategy: publish a comparison post—"LiteLLM vs Portkey vs Self-Hosted Gateway"—that targets the highest-intent keyword. Developers searching for gateway comparisons are actively evaluating tools. This single post can capture the entire early market.

Risk Assessment

This thesis is wrong if any of the following happens:

Risk 1: LiteLLM consolidates the market. If LiteLLM ships a polished dashboard and commercial tier within 6 months, your differentiation evaporates. Validation: monitor LiteLLM's GitHub releases and their hiring for product roles. If they hire a product designer, accelerate your timeline or pivot.

Risk 2: Cloudflare or Vercel makes gateways a free bundled feature. If gateway functionality becomes a checkbox in their platforms, standalone products face extinction. Validation: track their changelogs. Mitigation: focus on multi-platform support—they will always be biased toward their own ecosystem.

Risk 3: The market consolidates around direct SDK usage. If the major AI providers all adopt a universal API standard (unlikely but possible), the gateway's value drops to just caching and analytics. Validation: monitor OpenAI and Anthropic for cross-provider compatibility announcements.

Cheap validation before building: create a landing page for the commercial tier, drive traffic via GitHub and Reddit (r/selfhosted, r/LocalLLaMA), and measure signup intent. If 200+ developers join a waitlist in 30 days, build. If fewer than 50, walk away.

Action Plan

Today: Publish a GitHub issue on a popular open-source AI project proposing a gateway feature, or write a technical blog post comparing gateway options. Gauge community response. This costs zero dollars and two hours.

Week 1: Build the MVP proxy core (5–7 days). Deploy it for your own AI usage. If you spend $500 monthly on LLM APIs, the gateway should cut that to $200. Use your own savings as the first case study.

Month 1: Launch on GitHub with strong documentation and a Product Hunt submission. Target 500 GitHub stars. Post on Hacker News and r/selfhosted. Track which use cases resonate.

Month 3: If you have 50+ active GitHub users and 10+ developers asking for the commercial dashboard, build it. Price at $49/month and approach your first 20 users directly. If adoption is below 20 active users, reassess the positioning or pivot to a narrower use case.

The signal that confirms this opportunity: developers independently deploying your gateway in production and reporting cost savings. That is worth more than any market research.

Related Terms

LLM Observability — the practice of monitoring, tracing, and analyzing AI application behavior. Gateways are natural collection points for observability data, making these categories adjacent and complementary.

Model Routing / Model Selection — the emerging practice of dynamically choosing which AI model handles each request based on complexity, cost, and quality requirements. This is the killer feature for gateways, not just a side benefit.

Self-Hosted AI Infrastructure — the broader movement toward running AI workloads on your own infrastructure for cost, privacy, and compliance reasons. The gateway is the control plane for this movement, and its growth tracks the broader shift away from single-provider cloud dependency.