Decentralized RAG

What is it

Decentralized RAG is retrieval-augmented generation where the retrieval layer is not a single vector database sitting behind one API, but a peer-to-peer mesh of semantic indexes that route queries to whichever node holds the most relevant embeddings. Instead of every app shipping its own Pinecone or pgvector instance, nodes announce what they know, and a router (the ZeroRAG paper's "adaptive semantic peer routing") forwards each query to the peers most likely to answer it well.

Technically, the moving parts are: a local-first embedding store on each node, a routing layer that scores peer relevance from metadata and past hit rates, and a protocol for exchanging chunks without centralizing them. The MCP angle matters — the npm tag mcp-server signals that developers are already wrapping local RAG as Model Context Protocol servers, so an agent can call "search my docs" as a tool.

Business significance: this is the retrieval equivalent of what Git did to version control and BitTorrent did to file distribution. It lets teams keep sensitive corpora on their own machines while still composing a shared knowledge layer — a direct answer to the compliance objection that kills most enterprise RAG deals. For indie developers, the opening is infrastructure and tooling, not the model itself.

Why now

Three forces converge in late 2026. First, MCP has become the de facto standard for connecting agents to tools — Anthropic shipped it in late 2024, and by 2026 every serious coding agent and IDE speaks it. That gives decentralized RAG a ready-made transport and a distribution channel: ship an MCP server, and Claude Desktop, Cursor, and VS Code users can install you in one config line.

Second, the cost curve of local embeddings collapsed. Small models like nomic-embed-text and bge-small run on a laptop CPU at acceptable latency, and quantization means a 10M-chunk corpus fits on a 16GB machine. The technical barrier to being a "peer" dropped to near zero.

Third, enterprise procurement has hardened around data residency. Regulated buyers (health, legal, finance) increasingly refuse to ship documents to a third-party vector DB. Local-first RAG stopped being a hobbyist preference and became a checkbox on security reviews.

The timing is narrow. This is the window where the protocol is standardized but the tooling is not. Twelve months from now, either a big vendor (Pinecone, Weaviate, or a cloud provider) ships a managed federated offering, or the category consolidates into a feature of existing MCP servers. The 100% growth rate and nascent stage in the source data reflect exactly this pre-consolidation moment.

Market Evidence

The signal is thin but directionally clean: 2 independent sources (one arXiv paper, one npm package), 2 total mentions, 100% growth rate, stage "nascent," trend score 61/100. Read this honestly — two mentions is not demand, it is a leading indicator. The arXiv paper establishes that the routing problem is academically interesting and solvable; the npm mcp-server tag establishes that at least one developer shipped something real rather than just writing.

What makes this more than noise is the type of signal. NPM packages are shipped by people who intend to use them, not by people writing thought-leadership. A single MCP server with a vector-search tag is a weak signal alone, but it sits inside a much larger, measurable wave: the MCP ecosystem has thousands of servers, and "local RAG over my notes/codebase" is one of the most-cloned categories. The decentralized variant is the natural next step once users have three local indexes and want to query them together.

Compare to how other categories looked at this stage. Early vector DB tooling (2021) had similarly sparse academic-plus-repo signals before the 2023 explosion. That is the pattern to watch: academic formalization plus one working repo, then a 6-12 month gap before commercial tooling.

Verdict: real but early. The demand is latent — developers are already building local RAG and hitting the "how do I combine sources" wall. They just have not named the problem "decentralized RAG" yet. Your job is to name it for them.

Who's Behind It

No single whale owns this yet — that is the opportunity. The academic side is driven by the ZeroRAG authors, whose contribution is the adaptive semantic peer routing algorithm: instead of broadcasting queries to all peers (expensive) or using a static hash (dumb), they learn per-query routing weights. That is the intellectual core, and it is published, so nobody has a patent moat.

The infrastructure side is the MCP ecosystem: Anthropic as the protocol steward, and the long tail of server authors on npm and GitHub. Adjacent whales who could enter: Pinecone and Weaviate (managed vector DBs that would love a federated story), LangChain/LlamaIndex (framework layer, already shipping local loaders), and Ollama (owns the local-model developer mindshare — a natural home for a peer router).

The community is the local-first software movement (Ink & Switch lineage, CRDT folks like Automerge and Yjs) plus the self-hosted crowd on r/LocalLLaMA. These are the people who will star your repo and file your first bug reports. They are also cheap to reach: they congregate in a handful of subreddits and Discords, and they respond to working code, not marketing.

Competitive dynamic to watch: if Ollama or LangChain ships a "federated retrieval" flag, your standalone product becomes a feature. Move fast on the tooling layer they will not bother with — observability, routing dashboards, and the boring enterprise bits.

TAM & Market Size

The buyers are developers and small teams running RAG in production who have hit the multi-source wall. Bottom-up: the MCP ecosystem is the best proxy. If even 5% of the tens of thousands of MCP server authors and their users need to federate indexes, that is a few thousand paying seats — enough for a $1-3M ARR indie business, not enough for a VC-scale outcome. Be honest about that ceiling.

Segments, in order of willingness to pay:

1. Regulated SMBs (law firms, clinics, boutique financial advisors) — 5-50 seats, cannot send docs to a cloud vector DB, will pay $20-50/seat/month for a compliant federated layer. Highest price tolerance.
2. Indie devs and small SaaS teams — 1-5 seats, will pay $15-29/month for a tool that saves them building routing themselves. Self-serve, low CAC.
3. Enterprise platform teams — 100+ seats, but 9-12 month sales cycles and security reviews. Ignore until you have 50 SMB customers.

The 0/100 opportunity and demand scores in the source data are artifacts of the nascent stage, not a verdict on the market — the scoring model has too few data points. Treat them as "unmeasured," not "zero." The real number to track is npm weekly downloads of the existing MCP RAG servers; that is your demand thermometer.

Price tolerance realistically sits at $19-49/month for self-serve and $15-30/seat for team plans. Above $99/month you are selling to buyers who expect SOC 2.

Competitive Landscape

Direct competitors: essentially none shipping a productized decentralized RAG. The ZeroRAG paper is research, not a company. The npm MCP servers are single-node local RAG — they do not route across peers. Competition score of 0/100 reflects an empty field.

Indirect competitors, and how they lose:

• Pinecone / Weaviate / Qdrant Cloud — centralized, managed, and priced per vector. They win on convenience and lose on data residency. If they add federation, they will do it as an enterprise tier at $500+/month, leaving the SMB and indie segment open.
• LlamaIndex / LangChain — frameworks, not products. They give you the plumbing but not the router, the dashboard, or the ops. Developers using them still have to build peer discovery themselves.
• Ollama + Open WebUI — owns local-model distribution. If they ship a peer index, they commoditize the base layer. Your defense: be the neutral layer that works across Ollama, LM Studio, and hosted models.

Differentiation opportunity: be the routing and observability layer, not another vector store. Ship a dashboard that shows which peer answered which query, hit rates, and latency — the thing every team debugging a federated system will desperately want. That is unglamorous, sticky, and exactly what frameworks will not build.

Time horizon: Big Tech entry is plausible within 12-18 months. You have roughly three quarters to establish a niche and a customer base before the feature gets absorbed.

Business Model

Recommendation: freemium self-serve SaaS with a per-seat team tier, plus a usage-based API for the routing layer.

Why this fits: the buyer is a developer who wants to try before buying. A free tier of 3 peers and 50K chunks lets them validate on a real project in an afternoon. The upgrade trigger is predictable — they add a fourth peer, or they need the routing dashboard, or they need SSO. That is a natural paywall, not an artificial one.

Pricing:

• Free — 3 peers, 50K chunks, community support. Drives npm installs and word of mouth.
• Pro — $29/month per user, unlimited peers, routing dashboard, 1M chunks. Anchor here for indie devs and small teams.
• Team — $19/seat/month at 5+ seats, SSO, audit log, priority support. The per-seat drop rewards expansion.
• API — $0.50 per 1K routed queries beyond 100K/month. Captures high-volume agent workloads without punishing small users.

12-month forecast (conservative / base / optimistic): 40 paying accounts at ~$35 blended = $1.4K MRR; 150 accounts = $5.2K MRR; 400 accounts = $14K MRR. These are realistic indie numbers, not hockey sticks.

CAC: near zero for the first 100 customers if you launch on npm, GitHub, and r/LocalLLaMA with working code. Paid acquisition is a trap here — the audience is too small for ad economics. Payback on a $29/month customer acquired via content is under one month; that is the whole thesis.

MVP Blueprint

Build the smallest thing that proves routing works across two machines. Seven days, hard stop.

Day 1-2 — Core peer node. TypeScript, Node 20, using @modelcontextprotocol/sdk to expose a search tool. Local embeddings via nomic-embed-text through Ollama, stored in a SQLite file with sqlite-vec for the vector index. Each node exposes an HTTP endpoint returning its top-k chunks for a query plus a self-description (corpus topic, chunk count, model used).

Day 3-4 — Router. The differentiator. A small service that takes a query, scores each registered peer using (a) embedding similarity between the query and the peer's corpus description, and (b) historical hit rate per peer, then fans out to the top 2 peers only. Log every routing decision to SQLite. This is where the ZeroRAG paper's insight lives — keep it simple, no ML training in v1.

Day 5 — The dashboard. One page: peers online, queries routed, per-peer hit rate and latency, recent routing decisions with the chunks returned. This is your sales artifact. Make it look good.

Day 6 — Install path. npx decentralized-rag init scaffolds a node, writes the MCP config snippet for Claude Desktop and Cursor. Publish to npm.

Day 7 — Landing page and demo video. A 90-second screen recording of two laptops on one Wi-Fi, query hitting both, dashboard lighting up.

Stack: TypeScript, Node, SQLite + sqlite-vec, Ollama embeddings, MCP SDK, Next.js for the dashboard. No cloud dependency in v1 — that is the point. Cut: auth, billing, multi-tenant, any hosted offering. Ship the local proof first.

Commercial Opportunities

1. The federated RAG router as a self-hosted product. Target: platform teams at 20-200 person companies running internal knowledge search. They have three indexes (Confluence, codebase, shared drive) and no way to query them together without shipping everything to one vendor. Sell a Docker image with the router and dashboard for $499/month flat. Why it beats alternatives: cheaper than consolidating into a managed vector DB, and it satisfies the security team without a 6-month review.

2. Managed routing API for agent builders. Target: indie developers building MCP-based agents who want cross-source retrieval without running infrastructure. You host the router, they run their own nodes or use yours. Price at $0.50/1K queries with a $29 floor. Why it beats alternatives: LangChain gives them plumbing but not routing; you give them a single endpoint that already knows which of their sources to hit.

3. Compliance-first federated search for regulated SMBs. Target: 10-50 person law firms and clinics. Package the router with a pre-built connector set (SharePoint, Clio, Doximity exports) and a data-residency guarantee. Price at $25/seat/month. Why it beats alternatives: general vector DBs cannot promise "your documents never leave your network," and that promise is worth more than the feature list to this buyer.

Each direction is a real business at indie scale. Direction 1 is the fastest to revenue; direction 3 has the highest price tolerance but the longest sales cycle.

Product Ideas

🥇 PeerRAG — "Query all your local knowledge bases as one, without moving a single document." Target: developers and small teams with 2-5 separate RAG indexes (notes, codebase, docs, email). Why now: MCP made local RAG trivial, so the multi-source problem is the next wall, and nobody has shipped a router for it. Ship as an MCP server plus dashboard, $29/month Pro tier. This is the MVP from the blueprint, productized.

🥈 RouteLens — "Observability for federated retrieval — see which peer answered, and why." Target: teams already running a multi-node RAG setup who are debugging bad answers. Why now: every federated system becomes unmanageable without routing visibility, and the frameworks will not build it. Price at $19/month per workspace, or bundle free with PeerRAG to drive adoption. This is the sticky wedge — once a team's debugging workflow depends on your dashboard, switching costs are real.

🥉 Federated Docs API — "One API endpoint that searches across your customers' document stores, without ingesting them." Target: vertical SaaS vendors (legal tech, healthcare scheduling) whose customers refuse to upload documents to the vendor's cloud. Why now: data residency objections are killing enterprise deals, and a routing API lets the vendor say "we never store your docs." Price at $0.50/1K queries, $299/month minimum. Longer sales cycle, but the buyer has budget and a burning problem.

Priority order reflects time-to-revenue. Ship 🥇 first, use it to validate routing demand, then layer 🥈 as the retention product.

SEO Opportunity

Search volume for "decentralized RAG" is effectively zero today — that is the point. You are not competing for existing demand; you are defining the term. SEO difficulty of 0/100 means the first credible article ranks immediately.

Long-tail keywords to own: "federated RAG MCP server," "local RAG multiple sources," "MCP vector search server," "peer to peer vector database," "ZeroRAG implementation." Each has low volume but high intent from developers who are actively building.

Content strategy: publish the definitive "what is decentralized RAG" explainer with a working demo repo, then a tutorial for each MCP client (Claude Desktop, Cursor, VS Code). Tutorials rank and convert; explainers get linked. Write the tutorial first.

Risk Assessment

Risk 1 — Protocol absorption (technical). MCP or a major client adds native multi-server routing, making your router redundant. Likelihood: moderate. Mitigation: stay client-agnostic and own the dashboard, which is a product concern, not a protocol concern.

Risk 2 — The demand never materializes (market). Developers keep using one big centralized index because it is simpler, and "decentralized" stays a hobbyist preference. Likelihood: real. This is the biggest threat. Validation: watch npm downloads of existing local RAG MCP servers for 60 days — if they are flat, the multi-source pain is not being felt yet.

Risk 3 — You build infrastructure, not a product (execution). The routing algorithm is fun to build and easy to over-engineer. The trap is shipping a library instead of a product with a dashboard, install path, and pricing page.

Cheap validation before building: post the two-laptop demo video to r/LocalLLaMA and the MCP Discord, and count how many people ask "how do I install this." If fewer than 20 ask in a week, the demand is not there. Walk away if the existing MCP RAG servers show no download growth and no one asks for multi-source. Revisit in six months.

Action Plan

Today: Clone the MCP SDK, scaffold a single-node local RAG server with Ollama embeddings and sqlite-vec, and get it answering queries in Claude Desktop. Two hours of work. This proves the base layer.

Week 1: Build the router and the two-laptop demo. Record the 90-second video. Publish the repo to GitHub with a clear README and the npm package. Post to r/LocalLLaMA, the MCP Discord, and Hacker News ("Show HN: Route RAG queries across machines with MCP"). Goal: 50 GitHub stars and 20 installs.

Month 1: Ship the dashboard. Add the npx init installer. Launch the landing page with the $29 Pro tier and a free tier. Goal: 10 paying customers and 100 npm weekly downloads. Talk to every paying customer on a call — you need to hear the exact words they use for the problem.

Month 3: If you have 30+ paying customers and growing downloads, build the Team tier and SSO, and start the first regulated-SMB pilot. If you have fewer than 10 paying customers, stop and reassess — the signal is not confirming, and the 12-18 month Big Tech window is closing.

The single metric that decides everything: paying accounts at month 3. Under 10, walk. Over 30, double down.

Related Terms

MCP (Model Context Protocol) — the transport layer that makes decentralized RAG installable in one config line. Every peer node is an