Small Decision Models

What is it

Small Decision Models (SDMs) are compact AI models — typically under 1 billion parameters — trained to make one narrow decision extremely well, extremely fast, and often on-device. Think of Jeff's 0.8B home-trained model that returns decisions in roughly 30 milliseconds, laya's non-autoregressive "System 1" decision engine, and MicroLLM Lab's in-browser tiny models. Instead of asking a giant LLM to reason through everything, you train a small specialist that answers a single question: approve or decline, route or escalate, buy or skip.

The business significance is the cost curve. A 0.8B model can run on a laptop, a browser, or a $5/month VPS. That collapses inference cost from dollars-per-thousand-calls to fractions of a cent, removes the latency tax of API round-trips, and sidesteps data-privacy objections because nothing leaves the device. For indie developers, this is the first time "AI-powered" can be a feature you ship profitably at $9/month — not a loss leader subsidized by venture capital.

Why now

Three forces converged in 2026. First, the open-weight ecosystem matured: sub-1B models (Qwen3-0.6B, SmolLM2, Gemma 3n) now hit usable accuracy on classification, routing, and scoring tasks that previously demanded 7B+ models. Second, tooling got trivial — llama.cpp, ONNX Runtime, and WebGPU mean a solo dev can fine-tune and deploy in an afternoon, not a quarter.

Third, and most important, the economics of frontier APIs stopped making sense for high-volume, low-complexity decisions. If you're making 5 million "is this spam?" or "should I page on-call?" calls a month, paying $0.15-$3 per million tokens to a frontier model is absurd when a 30ms local model does it at 98% of the accuracy.

The HN thread (story_49882781) landing on the front page is the tell: practitioners are sharing home-trained decision models because the pain is real and the solution is now cheap. This wasn't true in 2024 — the small models weren't good enough — and it may not be the story in 2028 if frontier models get cheap enough to make specialization pointless. The window is now.

Market Evidence

The signal is early but clean: 2 independent sources (Hacker News and GitHub), 3 total mentions, a 100% growth rate, and stage classified as "nascent." A trend score of 62/100 with only 3 mentions means the topic is punching above its volume — the people who encounter it engage hard.

Read this honestly: 3 mentions is not a market. It's a leading indicator. Compare it to where "local-first software" sat in 2019 or "RAG" sat in early 2023 — a handful of front-page threads before an explosion. The 100% growth rate is trivially true at low base counts, so don't over-index on it.

What makes this credible rather than hype is the nature of the mentions: they're technical, specific, and reproducible (a 0.8B model, a named architecture, in-browser inference). Hype looks like "AGI next year." Demand looks like "here's my config and my latency numbers." This is the second kind.

Verdict: real, early, and under-served. The opportunity score of 0/100 reflects that no one has productized it yet — which is precisely why it's interesting. You have roughly 6-12 months before this becomes a crowded category.

Who's Behind It

There are no whales yet — and that's the point. The drivers are individual builders: "logicallee" (the HN author), Jeff with his 0.8B home-trained model, the laya team pushing non-autoregressive System 1 decision engines, and MicroLLM Lab shipping in-browser tiny models. These are indie hackers and small research groups, not OpenAI or Anthropic.

The surrounding infrastructure ecosystem is the real whale-adjacent layer: Hugging Face (model hosting and fine-tuning), llama.cpp and Ollama (local inference), and the browser vendors pushing WebGPU. They won't build your product, but they'll keep lowering your cost floor.

Competitive dynamics are wide open. No incumbent owns "small decision models" as a category. Big Tech is distracted optimizing frontier models — a 0.8B specialist is beneath their roadmap. That leaves a clean lane for indie developers to define the category, own the terminology, and build the default tooling before anyone notices.

TAM & Market Size

The buyers are developers and ops teams drowning in high-volume, low-stakes decisions. Concretely: SaaS companies doing content moderation, fraud screening, lead scoring, support ticket routing, and on-call alert triage. That's tens of thousands of companies, each making millions of repetitive classification calls monthly.

Price tolerance is real but modest. These buyers already pay for API calls (often $200-$2,000/month) or cloud inference. A tool that cuts that spend 80% while improving latency has obvious ROI. Expect willingness to pay $29-$199/month for a developer tool, or usage-based pricing at $0.50-$2 per 100K decisions.

The honest caveat: the opportunity score is 0/100 and demand score is 0/100, meaning the market is unproven. You're not entering a validated category — you're betting that "cheap local decisions" becomes a budget line item. The TAM is large in theory (every app with a classifier) but currently unmeasured. Treat early revenue as validation, not scale.

Competitive Landscape

Direct competitors are essentially nonexistent as products. The closest are generic ML platforms (Hugging Face Inference Endpoints, Replicate, Modal) that let you deploy small models but don't specialize in decision-making workflows. They're infrastructure, not solutions — you still have to build the training loop, the decision logic, and the monitoring.

Indirect competitors are the frontier APIs themselves: OpenAI, Anthropic, and Google. Their weakness is exactly SDMs' strength — cost and latency at volume. Their strength is zero-setup convenience. Your wedge is the high-volume, latency-sensitive, privacy-conscious segment they can't serve economically.

The gap: nobody offers a turnkey "train a decision model from your labeled examples in 10 minutes, deploy to edge, monitor drift" product. That's the opening.

If Big Tech enters — say, OpenAI ships a "fine-tune a tiny decision model" button — you have maybe 12-18 months. But they historically ignore sub-1B use cases. Your defense is vertical depth: own one niche (e.g., fraud decisions for fintech) so completely that a generic tool can't match it. Competition score 0/100 means the field is empty; move fast to stay ahead of the copycats who will follow the front-page signal.

Business Model

Go with usage-based SaaS plus a free tier. Freemium is mandatory here because developers must test accuracy on their own data before trusting a decision model — a paywall before validation kills conversion. Free tier: 10K decisions/month, 1 model, community support. Paid tiers scale on volume and features.

Suggested pricing:

• Starter — $29/month: 500K decisions, 3 models, drift monitoring, email support.
• Growth — $99/month: 5M decisions, 10 models, A/B testing, priority support.
• Scale — $299/month: 50M decisions, unlimited models, on-prem/edge deploy, SLA.

Rationale: this undercuts frontier-API spend by 70-90% while staying above the "too cheap to be serious" line for B2B buyers.

12-month forecast (assuming 6-month build-to-traction):

• Conservative: 80 paying customers, avg $60/mo → ~$58K ARR.
• Base: 250 customers, avg $75/mo → ~$225K ARR.
• Optimistic: 700 customers, avg $90/mo → ~$756K ARR.

CAC estimate: $150-$400 via developer content, HN, and SEO. At $75/mo average and ~85% gross margin, payback lands at 3-6 months — healthy for a dev tool. The model works because marginal inference cost is near zero once models run on customer hardware or your cheap CPU fleet.

MVP Blueprint

Ship in 5-7 days. Cut everything that isn't "upload data → get a working decision model."

Core features only:

1. Upload labeled examples (CSV/JSONL) — binary or multi-class decisions.
2. One-click fine-tune of a sub-1B base model (Qwen3-0.6B or SmolLM2) via LoRA.
3. Accuracy report — precision/recall on a held-out split, so users trust it.
4. REST API endpoint — POST input, get decision + confidence in <50ms.
5. Usage dashboard — decisions made, latency, drift alert.

Tech stack: Python + FastAPI backend, llama.cpp or ONNX Runtime for inference, a hosted GPU for fine-tuning (Modal or Replicate), Postgres for metadata, Next.js for the frontend. Deploy inference on CPU (cheap) with an optional WebGPU in-browser mode as a differentiator.

Fastest path to launch: skip auth complexity (use magic links), skip billing on day one (Stripe after first 10 users), skip multi-tenancy polish. The goal is a working demo you can post to Hacker News with real latency numbers — that's your marketing.

Suggested product types align: SaaS for the dashboard, a CLI Tool for local runs, and an API for production integration. Start with the API; it's the stickiest surface.

Commercial Opportunities

1. Fraud & risk decision API for fintech. Target: fintech startups and payment processors making millions of approve/decline calls. Expected revenue: $5K-$30K/month. Why it beats alternatives: fraud decisions are latency-critical and privacy-sensitive — exactly where local 30ms models crush API round-trips, and where sending data to a third party is a compliance headache.

2. Support ticket routing & triage tool. Target: SaaS support teams with 10K+ tickets/month. Expected revenue: $3K-$15K/month. Why it wins: routing is a narrow classification task, cheap to train, and the ROI (agent time saved) is trivially demonstrable in a pilot.

3. On-call alert noise reduction. Target: DevOps/SRE teams at mid-size companies. Expected revenue: $2K-$12K/month. Why it beats alternatives: alert fatigue is a known pain, decisions are binary (page / don't page), and a 30ms local model can run inside the monitoring pipeline without adding latency.

Each direction beats "generic AI platform" because it owns a specific, painful, high-volume decision and can prove ROI in one pilot.

Product Ideas

🥇 DecisionKit — "Train a decision model from your spreadsheet in 10 minutes." Target: indie developers and small SaaS teams who need a classifier but can't afford ML engineers. Why now: sub-1B models just became accurate enough, and no turnkey tool exists.

🥈 EdgeRouter API — "30ms decisions at 1/100th the cost of frontier APIs." Target: high-volume API consumers (moderation, scoring, routing). Why now: the cost gap versus OpenAI/Anthropic is at its widest, and the front-page signal proves demand.

🥉 DriftGuard — "Monitor and retrain your decision models automatically." Target: teams already running small models who fear silent accuracy decay. Why now: as SDMs proliferate, drift monitoring becomes the obvious second-order need — and it's a natural upsell on the first two products.

Ranking logic: DecisionKit is the wedge (broadest, easiest to sell), EdgeRouter is the monetization engine (usage-based, sticky), DriftGuard is the retention layer (expands accounts). Build in that order.

SEO Opportunity

Search interest for "small language models," "on-device inference," and "fine-tune small model" is climbing but still low-volume — classic early-category SEO where you can rank with modest effort. SEO difficulty reads 0/100, meaning near-zero competition for the exact terms.

Target long-tail keywords:

• "train small decision model python"
• "sub-1B model fine-tune tutorial"
• "on-device classification model API"
• "cheaper alternative to OpenAI API for classification"
• "30ms inference local model"

Content strategy tip: publish reproducible benchmarks — "I replaced GPT-4 with a 0.8B model for ticket routing and cut costs 94%." Data-driven posts earn backlinks and rank fast in a category with no established authorities. Own the term "Small Decision Models" before someone else defines it.

Risk Assessment

Tech risk: Sub-1B models may plateau in accuracy on harder decisions, capping your market to trivial classification. Mitigation: validate on real customer data before scaling.

Market risk: Frontier API prices could collapse, erasing your cost advantage. If OpenAI drops to $0.01/million tokens, the "cheaper" pitch dies. Mitigation: lean on latency and privacy, which price cuts don't fix.

Execution risk: You're building ML tooling, not a CRUD app — fine-tuning reliability is hard. Mitigation: start with a single base model and one decision type.

Cheap validation before building: post a landing page offering "train your decision model free" and collect emails. Run a manual concierge version for 5 prospects — train their models by hand. If they don't convert to paid pilots, the thesis is weak.

Walk away if: after 10 concierge pilots, fewer than 3 convert to paid, or accuracy on real data stays below 90%. That means the tech isn't ready or the pain isn't acute.

Action Plan

Today: Post a landing page ("Train a small decision model in 10 minutes — join the beta") and share it in the HN thread and relevant subreddits. Collect emails and note which use cases people mention.

Week 1: Manually train decision models for 5 respondents using their data. Measure accuracy and latency. This is your cheapest possible validation — no product, just results.

Month 1: If 3+ pilots show >90% accuracy and interest in paying, build the MVP from the blueprint. Ship the API first. Get 10 paying customers at $29/month.

Month 3: Reach $2K-$5K MRR, publish benchmark content, and pick one vertical (fintech, support, or DevOps) to go deep. Add drift monitoring as the retention hook.

The discipline: don't build until the concierge pilots prove accuracy and willingness to pay. The signal is early — 3 mentions — so validation must come from real users, not the trend score.

Related Terms

Small Language Models (SLMs) — the broader category SDMs live inside; SDMs are SLMs applied to a single decision. On-device / edge inference — the deployment pattern that makes SDMs cheap and private. Non-autoregressive models — laya's approach, generating a decision in one pass rather than token-by-token, which is why 30ms latency is possible.

Together they form a stack: SLMs provide the models, edge inference provides the economics, and non-autoregressive architectures provide the speed. Small Decision Models is the productized intersection — and the most commercially actionable of the three.