llms.txt

What is it

llms.txt is a proposed convention that places a Markdown file at the root of a website (yourdomain.com/llms.txt) containing a curated, structured index of the site's most important content — links, summaries, and context — written specifically for consumption by large language models rather than human browsers or traditional search crawlers. Think of it as robots.txt for the AI era: instead of telling bots what not to crawl, it tells AI systems what matters and how to interpret it. The format was popularized by the Answer.AI community in late 2024 and has since been picked up in developer circles worldwide.

The business significance is straightforward. As AI assistants (ChatGPT, Claude, Perplexity, Gemini) become the default way people discover products, services, and information, a site that is invisible to LLMs is effectively invisible to a growing share of buyers. llms.txt is the first attempt at a standard handshake between websites and AI retrieval systems. That creates a wedge for tooling: generators, validators, hosting, analytics, and monitoring — a whole infrastructure layer that doesn't exist yet.

Why now

Three forces converge in 2026. First, AI-referred traffic crossed a meaningful threshold — by mid-2025, Perplexity alone was fielding hundreds of millions of queries monthly, and ChatGPT's search product pushed into mainstream consumer behavior. Marketers who ignored this in 2024 are now being asked by their boards where their "AI traffic" is.

Second, the standard itself is gaining legitimacy. Answer.AI's proposal, plus adoption by documentation-heavy sites (Stripe-style developer docs, Mintlify-hosted sites, and several open-source projects), has moved llms.txt from a tweet to something crawlers actually check. Cloudflare and other CDN players have started discussing native support.

Third, and most tellingly, the Chinese developer community ran a real-world test: of 1,000 sites surveyed, only 30 had deployed llms.txt. That 3% adoption rate is exactly the kind of gap that precedes a tooling gold rush — the same pattern we saw with robots.txt validators, sitemap.xml generators, and Open Graph preview tools. The standard exists, awareness is rising, and almost nobody has implemented it. That's a timing window, not a permanent state.

Market Evidence

The hard data is thin but directional: 1 independent source, 2 total mentions, a 100% growth rate, and a "nascent" stage classification. The V2EX thread that surfaced this is a classic early-adopter signal — a technical community independently benchmarking real-world adoption and finding it near-zero. That's not hype; that's a measured gap.

Two mentions is not a market. Let's be honest about that. But the quality of the signal matters more than the count here. The 30/1000 finding is a concrete, reproducible number that any founder can verify by crawling a few thousand sites themselves. Compare this to a typical "trend" that's just VC-funded noise — llms.txt has an actual adoption metric attached, and that metric is embarrassing for everyone who claims to care about AI discoverability.

The 100% growth rate is a statistical artifact of going from 1 mention to 2, so discount it entirely. The real question is whether the 3% adoption rate is a floor or a ceiling. Given that AI traffic is rising, that major docs platforms are adding support, and that no dominant tooling exists yet, I'd bet on floor. But this is a "verify in a week" opportunity, not a "raise a seed round" one.

Who's Behind It

The originator is Answer.AI (Jeremy Howard's lab), which published the llms.txt proposal as a lightweight spec. The early adopters are developer-tooling companies with strong docs cultures: Mintlify (which added llms.txt generation to its docs platform), Anthropic-adjacent projects, and a long tail of open-source repos. In the Chinese ecosystem, the V2EX community is the visible testing ground, but the actual implementers are likely indie devs and small SaaS teams running their own docs.

The "whales" to watch are the docs platforms — Mintlify, Docusaurus, GitBook, ReadMe — because whoever ships llms.txt generation as a default feature owns the distribution. Cloudflare is the other whale: if it adds a one-click llms.txt toggle in its dashboard, the adoption rate jumps overnight and the standalone-tool window slams shut. Right now none of them have made it a headline feature, which is precisely why an indie tool can win the next 6-12 months.

TAM & Market Size

The buyers are anyone who runs a content-heavy website and cares about AI visibility: SaaS companies, developer-tool startups, agencies, e-commerce sites, publishers, and docs teams. Start with the narrowest, highest-intent segment: developer-tool and SaaS companies with public documentation. There are roughly 30,000-50,000 such companies globally, and a meaningful share already pay for SEO tooling ($99-$500/mo for Ahrefs, Semrush, or Clearscope).

Price tolerance: this is a "nice-to-have that becomes a must-have" product. Early adopters will pay $19-$49/mo for a generator + validator + monitoring dashboard. Agencies managing 20+ client sites will pay $99-$299/mo. The broader "any website owner" market is millions of sites, but most won't pay until AI traffic is a line item in their analytics — which is a 2027 problem.

The provided scores (opportunity 0/100, demand 0/100) reflect a data-poor snapshot, not a verdict. My read: the serviceable market today is small (maybe $2-5M ARR ceiling for a pure-play tool), but the option value is large if llms.txt becomes standard. Treat this as a cheap call option, not a sure thing.

Competitive Landscape

Almost nobody is here yet, which is both the opportunity and the warning. Current players: Mintlify bundles llms.txt generation into its docs hosting (but only for Mintlify customers). A handful of open-source GitHub scripts generate a basic llms.txt from a sitemap. Answer.AI's own reference tooling is minimal. There is no dedicated validator, no monitoring service, no analytics product, and no agency-facing dashboard.

That's the gap. The differentiation opportunity is the monitoring and analytics layer: not just "generate a file," but "prove that AI crawlers are actually reading it, and show what they're citing." That's a recurring-revenue product, not a one-time script.

Big Tech threat timeline: if Cloudflare ships a native toggle (plausible within 12 months), it kills the pure generator. If Google or OpenAI publish their own preferred format, it could obsolete llms.txt entirely. So you have roughly a 6-12 month window to build the monitoring/analytics moat before the commodity layer gets eaten. Competition score of 0/100 is accurate today — and won't be in a year.

Business Model

Recommended: freemium SaaS with usage-based tiers. The generator is a commodity (give it away free to drive top-of-funnel); the money is in validation, monitoring, and analytics. This fits because the pain is recurring — sites change, crawlers behave differently, and "is the AI reading my site?" is a question you answer monthly, not once.

Suggested pricing:

• Free: generate + validate one llms.txt, basic crawl check.
• Pro — $29/mo: 3 sites, daily crawler monitoring, AI-referral analytics, alerts when your file breaks.
• Agency — $149/mo: 25 sites, white-label reports, client dashboards, bulk regeneration.

Rationale: $29 sits below the "needs approval" threshold and above the "toy" threshold. Agencies are the highest-LTV segment and the fastest path to $10K MRR.

12-month forecast (assumes 6-month build-to-traction):

• Conservative: 150 Pro + 15 Agency = ~$6.5K MRR.
• Base: 500 Pro + 60 Agency = ~$23K MRR.
• Optimistic: 1,500 Pro + 200 Agency = ~$73K MRR.

CAC estimate: $40-$120 via content/SEO and developer communities. Payback: 2-4 months on Pro, under 1 month on Agency. The economics work if you keep churn under 5%/mo — which requires the monitoring product to be genuinely sticky.

MVP Blueprint

Scope: 5 days. Cut everything else.

Core features ONLY:

1. Generator: paste a URL (or upload a sitemap.xml) → auto-produce a valid llms.txt with title, summary, and curated links. Use an LLM to write the summaries.
2. Validator: check an existing llms.txt against the spec and flag errors.
3. Hosted endpoint: serve the file at yourdomain.com/llms.txt via a CNAME or a hosted subdomain, so non-technical users can deploy without touching their server.
4. Basic crawler log: show whether known AI user-agents (GPTBot, ClaudeBot, PerplexityBot) have hit the file in the last 7 days.

Tech stack: Next.js + Vercel for the app, Postgres (Supabase) for storage, a small edge function for the hosted endpoint and log ingestion, OpenAI/Claude API for summary generation. Total infra cost under $50/mo at launch.

Fastest path to launch: ship the generator + validator as a free tool first, collect emails, then gate monitoring behind the $29 tier. Don't build auth, teams, or billing until you have 50 free users. The dev-day estimate is 0 in the data because this is genuinely a weekend-to-one-week build — that's the point of the opportunity.

Commercial Opportunities

1. AI-visibility monitoring dashboard for SaaS marketing teams. Target: growth/marketing leads at 50-500 person SaaS companies. Expected revenue: $5K-$25K MRR within 12 months. Why it beats alternatives: SEO tools (Ahrefs, Semrush) don't track llms.txt or AI crawler behavior yet, so you're selling into an unclaimed budget line.

2. White-label llms.txt + AI-SEO service for agencies. Target: digital agencies managing 10-100 client sites. Expected revenue: $8K-$40K MRR. Why it beats alternatives: agencies need to show clients they're AI-ready; a white-label report is an easy upsell they'll resell at 3-5x. Higher LTV, lower churn than self-serve.

3. Open-source generator + paid cloud hosting. Target: indie devs and small docs sites. Expected revenue: $2K-$10K MRR. Why it beats alternatives: open-source drives distribution and trust; the paid tier is "we host it, monitor it, and alert you when it breaks." This is the bottom-up wedge that feeds opportunities 1 and 2.

Product Ideas

🥇 LLMReady — "Ship an llms.txt in 5 minutes and prove AI crawlers read it." Target: SaaS docs and marketing teams. Why now: 97% of sites haven't deployed it, and no monitoring product exists. This is the flagship.

🥈 AIVisibility Report — "See exactly how AI assistants cite your site — and where you're invisible." Target: agencies and content marketers. Why now: AI-referral traffic is real but unmeasured; agencies will pay for a client-facing report they can't get elsewhere.

🥉 llms.txt Validator API — "One API call tells you if a site is AI-readable." Target: SEO platforms, site builders, and CMS vendors who want to embed the check. Why now: the standard is stabilizing and platform vendors need a drop-in compliance check before they build their own. Lower ceiling, but a fast B2B2C distribution play.

Priority order reflects revenue speed and defensibility: the dashboard has recurring revenue and a data moat; the report monetizes agencies fast; the API is a land-grab that could be commoditized.

SEO Opportunity

Search interest in "llms.txt" is rising from near-zero, tracking the standard's adoption curve. SEO difficulty is essentially 0/100 — there's almost no competing content. Target long-tail keywords: "llms.txt generator," "how to create llms.txt," "llms.txt validator," "llms.txt vs robots.txt," "does ChatGPT read my website." Content strategy: publish the definitive how-to guide plus a free generator tool on the same page — the tool earns links, the guide earns rankings. Move now; this window closes the moment a big SEO tool publishes a pillar page.

Risk Assessment

When the thesis is wrong: if AI vendors converge on a different mechanism (server-side structured data, direct API feeds, or proprietary crawlers that ignore llms.txt), the standard dies and so does the tooling market. This is the single biggest risk.

Top 3 risks:

1. Standard risk (high): llms.txt never gets official backing from OpenAI, Google, or Anthropic. Mitigation: build the monitoring layer, which is valuable regardless of format — track AI crawlers, not just one file.
2. Platform risk (high): Cloudflare, Vercel, or Mintlify ship native support and eat the generator. Mitigation: own the analytics and agency workflow, which platforms won't build.
3. Demand risk (medium): site owners don't care until AI traffic is measurable in their analytics. Mitigation: lead with the "here's your AI traffic" report, not the file.

Cheap validation: spend $200 on a landing page + Google Ads targeting "llms.txt generator" for one week. If you get 100+ signups and 5+ people asking to pay, build. If not, walk away. Walk-away signal: if adoption stays under 5% after a major platform ships native support, the market isn't forming.

Action Plan

Today: crawl 2,000 sites (use a public list like the Tranco top sites) and verify the 30/1000 adoption finding yourself. Publish the raw data as a blog post — it's instant credibility and SEO.

This week (low-cost validation): build the free generator + validator as a single-page tool. Launch on V2EX, Hacker News, and r/SaaS. Measure signups and inbound "can you monitor this for me?" requests. That request is your paid product.

If signal confirms: add the hosted endpoint and 7-day crawler log, gate monitoring at $29/mo, and start cold-emailing agencies with a free AI-visibility audit of their own site.

Timeline:

• Week 1: data post live, free tool shipped, 100+ signups.
• Month 1: paid monitoring tier live, first 20 paying customers, $500+ MRR.
• Month 3: agency tier launched, 100+ paying customers, $3K-$5K MRR, inbound from platform vendors about the API.

Related Terms

GEO (Generative Engine Optimization) — the broader discipline of optimizing content for AI answer engines. llms.txt is one tactical implementation of GEO; the monitoring product above is a GEO analytics play.

AI crawler analytics — tracking GPTBot, ClaudeBot, and PerplexityBot behavior. This is the natural evolution of llms.txt tooling and the more durable business.

Structured data for LLMs — schema.org extensions and emerging JSON formats aimed at machine readers. If llms.txt stalls, this is where the standard fight moves next — and where your monitoring layer should already be watching.