Self-hosted LLM vs API in 2026: cost, latency, and compliance decision matrix

The slide appears near the end of the quarterly business review. It is the kind of slide that gets no fanfare — a simple line chart, top-right corner trending upward at an angle that is, charitably, ambitious. The label on the Y-axis reads "API Spend (USD)." The X-axis covers January through May. The final data point is $14,200.

The CTO does not say anything for a moment. Someone on the finance team has already highlighted it yellow in the shared deck. "This is just model calls?" she asks.

"This is just model calls," the CTO confirms.

The product has grown. More users are generating content. More pipelines are running classification, summarization, and structured extraction jobs. None of these are wasteful — they are working features, used by paying customers, delivering value. The problem is that the API bill scales linearly with usage, and the usage trajectory has no natural ceiling.

The CTO already knows what the next thirty minutes of conversation will produce: a mandate to "look into self-hosting." This is how every self-hosting conversation in the LLM space begins — not with a clear evaluation framework, but with a line on a chart and a vague directive.

What follows that directive is often a sequence of expensive mistakes. The team researches GPU rental, finds that an 8×H100 node costs roughly $16,000 per month from a reputable provider, computes that this is more than their current API bill, and concludes self-hosting is not worth it. Or they go the other direction: they stand up an Ollama server on a rented GPU, run some benchmarks, see that costs look lower, and make the switch — only to discover six months later that they have hired a third of an SRE to keep the thing alive, and the total cost of ownership is materially higher than what they were paying before.

Neither outcome reflects bad intentions. Both reflect the absence of a structured decision framework. This post is that framework: a decision matrix for the API-vs-self-hosted question, built from the actual numbers, the actual failure modes, and three case studies in which each answer turned out to be the right one for different reasons.

What's actually happening

The LLM deployment landscape in 2026 has resolved into three distinct operating models, each with its own economics, latency profile, compliance posture, and operational burden. Understanding the tradeoffs requires being precise about all five dimensions simultaneously, because an analysis that optimizes for cost alone will almost always reach the wrong answer.

The three operating models

API-only is the default. You call Anthropic, OpenAI, Google, or Mistral's hosted endpoints, pay per token, and own none of the infrastructure. The economics are pure variable cost: the bill scales with token volume, you get the latest model the moment the provider ships it, and the ops burden is nearly zero. The tradeoffs are equally clear: you have no control over latency (you are on a shared endpoint with the rest of the world), you have no control over where your data goes (it traverses the provider's network and, depending on your agreement, may be retained for training), and your unit economics look unfavorable once you reach significant volume.

Self-hosted means you run inference on your own compute — either GPUs you own or dedicated GPU instances you rent. You serve open-weight models (Qwen 3, Llama 4, Mistral) via inference servers like vLLM, SGLang, or TGI. The token cost drops dramatically at high throughput. You control data residency. You control latency. You also own all of the operational complexity: GPU procurement, inference server configuration and patching, model updates, capacity planning, and the on-call rotation that none of the cost analyses remember to include.

Hybrid is the model that naive cost analyses miss and experienced platform engineers increasingly reach for. You route incoming requests by complexity and cost-sensitivity. Hard requests — multi-step reasoning, long-context synthesis, code generation requiring high correctness — go to the API. Easy requests — intent classification, routing decisions, short-form extraction, embedding generation — go to a self-hosted small model. The small model is cheap to run and fast enough that the latency overhead of the routing layer is invisible. You capture 60–80% of your token volume on the cheap path, keep frontier-quality outputs where they matter, and limit your self-hosted footprint to modest, highly utilized hardware.

The cost math

As of June 2026, Anthropic Claude Sonnet 4.6 is priced at approximately $3.00 per million input tokens and $15.00 per million output tokens (anthropic.com/pricing). OpenAI GPT-5 and Google Gemini 2.5 Pro sit in a similar tier, with slight variation by context length and task type. These are the models you reach for when you need frontier quality: they are updated frequently, they support long context natively, and they have broad multilingual and multimodal capability.

A self-hosted deployment on rented dedicated hardware carries a different cost structure. Using Lambda Labs pricing (lambdalabs.com) as a reference point for June 2026:

• A single H100 SXM5 instance runs approximately $2.80 per hour, or roughly $2,000 per month at continuous operation.
• An 8×H100 node (the minimum for serving a 70B-parameter model at meaningful throughput) runs approximately $16,000 per month.

For throughput on Qwen 3-70B via vLLM with continuous batching, expect 1,500–3,500 output tokens per second across a well-tuned batch depending on request mix and sequence length. At the midpoint — call it 2,000 tokens per second — running 24×7 at 100% utilization produces approximately 5.2 billion output tokens per month. The GPU cost per million output tokens at that utilization is roughly $3.10 (($16,000 ÷ 5,200) × 1,000). That is competitive with Sonnet 4.6 output pricing only if you can sustain near-100% utilization, which in practice almost no workload does.

The figure that routinely goes missing from self-hosted cost analyses is the operational labor. A production inference cluster requires monitoring, patching, on-call response, and model update management. A conservative estimate for a two-GPU-node setup managed by an existing SRE team: 0.5 FTE at a fully-loaded senior SRE cost of approximately $160,000 per year, or $6,700 per month. That single line item changes the economics of most self-hosted deployments.

The all-in monthly cost of an 8×H100 self-hosted cluster — hardware rental plus ops labor plus networking and storage — lands in the range of $22,000 to $28,000 per month. Against an API bill of $14,000 per month growing at a moderate rate, the immediate case for full self-hosting is weak.

The crossover math, roughly:

• Under $5,000/month API spend: API wins decisively. No self-hosted option pays off at this volume. Engineering time is better spent on product.
• $5,000–$20,000/month: Hybrid is usually the right answer. Route high-volume, low-complexity tasks to a small self-hosted model. API for everything requiring frontier quality.
• $20,000–$50,000/month: Self-hosting starts to win on cost for the right workloads. The ops burden is real, but the savings justify it if you can sustain utilization above 60%.
• Over $50,000/month: Self-hosted dominantly wins on cost. At this volume you have enough scale to run multiple model variants, keep utilization high, and amortize SRE cost across a larger workload.

These are rough thresholds, not bright lines. They shift with your workload's output-to-input ratio (high output volume amplifies cost savings), your ability to staff GPU infrastructure, and the specific models you are running.

The latency math

API latency has a floor set by network round-trips and shared endpoint scheduling. In June 2026, realistic time-to-first-token (TTFT) on Anthropic's US endpoints ranges from 200ms to 800ms for typical requests. Peak hours trend toward the higher end. Anthropic's enterprise agreements include SLA commitments, but P99 latency on shared endpoints under load is not guaranteed at the 200ms floor.

Self-hosted inference, with control over batching and endpoint proximity, typically achieves 50–200ms TTFT for Qwen 3-70B on an 8×H100 node with well-tuned vLLM configuration. For real-time user-facing applications — autocomplete, streaming chat, voice pipelines — that 3–4× latency difference is meaningful and user-visible. For batch processing, embeddings, and asynchronous pipelines, it is largely irrelevant.

Streaming throughput — tokens per second after the first token — is more similar between API and self-hosted than TTFT. Both are constrained by model size and hardware bandwidth. The self-hosted advantage is in TTFT and in tail latency: on shared infrastructure, P99 latency spikes occur when the provider's cluster is under load. Self-hosted P99 is determined by your own batching configuration and your own load patterns.

To understand why TTFT behaves differently from streaming throughput, it helps to think about what happens during those two phases. During prefill — the phase that produces the first token — the model processes all input tokens in parallel. On shared API infrastructure, this phase competes for compute with other users' prefill operations. The result is queuing: your prefill starts when the scheduler finds capacity for it. On a self-hosted endpoint where you control the request queue, prefill can begin as soon as the current batch has capacity. The gap widens during periods of high shared-endpoint load and narrows when provider infrastructure is under light utilization.

During the decode phase — the autoregressive token-by-token generation after the first token — GPU memory bandwidth, not compute, is the binding constraint. HBM3e memory on H100 SXM5 sustains approximately 3.35 TB/s bandwidth. This is a hardware constant that does not differ between API and self-hosted deployments running the same model. This is why streaming throughput looks similar across both: the physical constraint is the same silicon.

The practical latency decision criteria: if your product has a real-time component — streaming completions to a chat UI, voice pipeline response, interactive code suggestions — TTFT matters and self-hosted has a structural advantage at any scale. If your product is batch-oriented — nightly document processing, asynchronous report generation, background pipeline enrichment — the TTFT difference is invisible to users and the latency argument does not apply.

One intermediate case worth noting: batch inference APIs. Anthropic, OpenAI, and Google all offer batch inference endpoints that process jobs asynchronously over hours, at significantly lower cost than real-time endpoints. If your workload is latency-insensitive and volume is high, batch API endpoints often compete favorably with self-hosted even at mid-range spend levels. Anthropic's Batch API, for instance, prices at 50% of standard input/output rates with a 24-hour completion window — making it relevant for workloads like overnight document processing, bulk evaluation, or asynchronous content generation where sub-second response is not required.

The compliance reality

This is the dimension where the cost math becomes irrelevant. Certain deployment environments make API-only architectures legally or contractually impossible regardless of cost.

EU data residency: GDPR Article 44 and related transfer regulations create a complex landscape for sending personal data to non-EU-hosted endpoints. Anthropic and OpenAI have EU hosting options (Anthropic via AWS EU regions, OpenAI via Azure), but the contractual chain — data processing agreements, sub-processor lists, and the specific language around training data opt-out — requires careful legal review. For organizations for whom "no personal data may leave EU jurisdiction under any circumstances" is a hard requirement, self-hosting on EU-region GPU infrastructure is the default-safe choice.

HIPAA deployments: Anthropic and OpenAI both offer Business Associate Agreements (BAA) for HIPAA-covered workloads, but the scope of PHI that can transit those endpoints is constrained by the specific BAA terms and the organization's risk posture. Many healthcare technology organizations opt for self-hosted inference for any workload touching PHI, regardless of available BAAs, because the self-hosted model eliminates a category of third-party risk that their compliance team does not want to manage.

Air-gapped requirements: Defense, intelligence, and certain financial sector deployments operate on networks with no external connectivity. API calls are physically impossible. Self-hosted is the only option, full stop.

Customer contractual requirements: Enterprise SaaS contracts increasingly contain data handling clauses that specify "no customer data may be transmitted to third-party AI providers." If your customers are large enterprises with active procurement and legal review cycles, you will encounter this clause. Self-hosting lets you make that commitment cleanly.

SOC 2 Type II and audit trails: Some compliance frameworks require complete audit trails of where sensitive data was processed, including inference operations. Shared API endpoints complicate this: you can log your outbound requests, but you cannot audit the provider's internal processing pipeline. Self-hosted inference gives you complete control over the audit trail — you know exactly which machine processed which request, when, with what model weights, and what output was produced. For organizations building toward SOC 2 Type II certification with AI-heavy products, this audit coverage can simplify the certification process considerably.

The compliance argument has an asymmetry worth naming explicitly: a compliance constraint that blocks API usage does not require additional justification. It is a hard boundary. You do not need to build an ROI case for self-hosting when the alternative is not having the product at all. The useful analysis when compliance forces self-hosting is not "API versus self-hosted" — it is "what is the minimum self-hosted footprint that satisfies the constraint while keeping ops burden manageable?"

The model quality reality check

No self-hosted model in June 2026 matches GPT-5, Claude Opus 4.8, or Gemini 2.5 Ultra on frontier benchmarks. This is an important constraint to hold precisely, because it is commonly overstated in both directions.

Qwen 3-70B (huggingface.co/Qwen/Qwen3-70B) and Llama 4 (huggingface.co/meta-llama/Llama-4) sit approximately at the capability level of Claude Sonnet 4.5 or GPT-4o on most task-specific evaluations — strong reasoning, good code generation, solid multilingual performance. For the majority of production workloads — classification, extraction, summarization, question answering over structured context — the gap between a well-prompted Qwen 3-70B and a frontier model is small enough that careful evaluation often finds no material difference in downstream product metrics.

For genuinely hard tasks — multi-step mathematical reasoning, complex code synthesis, long-document synthesis requiring nuanced judgment — frontier models continue to lead by a measurable margin. The gap has closed significantly over the past 18 months, and it continues to close, but it has not disappeared.

Qwen 3-7B and Llama 4 8B (the small models you would run in a hybrid setup) are useful for classification, routing, embedding, simple Q&A, and short-form extraction. They are not suitable for tasks requiring long-context coherence or complex reasoning. This is not a failure — it is the appropriate application scope.

The practical implication: before committing to a self-hosted architecture, run an offline evaluation. Take 500–1000 representative production requests (sanitized of any PII), run them through your current API model and your candidate self-hosted model, and measure downstream task performance. Do not assume the models are equivalent. Measure it.

A useful framing for the evaluation: measure downstream product metrics, not model-level benchmarks. MMLU, HumanEval, and GSM8K are useful for broad model comparison but tell you nothing about whether your specific classification taxonomy is served equally well by both models, or whether your extraction prompt behaves the same against a different tokenizer. Instrument the evaluation against the actual task — classification accuracy, extraction F1, user-preference side-by-side comparison — and set a minimum acceptable threshold before committing to the model change. If the open-weight model passes the threshold on your actual task, you have a deployment-ready candidate. If it does not, you know before sinking engineering weeks into the infrastructure work.

One frequently overlooked dimension of model quality in self-hosted deployments is inference precision. Many cost analyses assume BF16 inference, which faithfully preserves model quality. Production self-hosted deployments often apply quantization — AWQ 4-bit, GPTQ 4-bit, or FP8 — to reduce VRAM footprint and improve throughput. Quantization degrades model quality by a measurable, workload-dependent amount. The throughput and cost improvements are real: AWQ 4-bit Qwen 3-70B on 4×H100 requires roughly 35GB of VRAM versus 140GB for BF16, allowing a smaller and cheaper hardware configuration. But the quality tradeoff is also real and must be evaluated. Include quantized models in your offline evaluation if you plan to use quantization in production.

The pattern

Four archetypal organizations face this decision differently. Understanding which pattern you match determines which framework section applies to you.

The fast-growing startup

You are pre-Series B, 15–30 engineers, and your API bill has crossed $5,000–$10,000 per month. It is growing faster than revenue. The conversation about self-hosting has started.

The critical question here is not cost — it is opportunity cost. Self-hosted GPU infrastructure requires someone to operate it. At a 20-person startup, there is no dedicated infra team. The engineers who would stand up vLLM, configure Kubernetes, write the monitoring, and own the on-call rotation are the same engineers who would otherwise be shipping product. The cost of a self-hosted cluster is not just the hardware line item; it is the product velocity you do not ship because two of your best engineers are debugging NCCL timeouts.

The right answer for most startups in this range is hybrid: spend one sprint adding a routing layer that classifies request complexity, sends easy requests to a small model on a single-GPU node (a single A100 or H100 costs $1.40–$2.00/hour on RunPod or vast.ai), and reserves the API for hard requests. That hybrid deployment reduces API spend by 50–70% on most workloads with two weeks of engineering effort and a single GPU node that does not require an SRE to babysit.

The mid-size SaaS

You are post-Series B, 80–200 engineers, an established product, and an API bill in the $20,000–$50,000/month range. You have an infra team. You have monitoring. You have on-call rotations already running for your database and application tier.

This is the zone where self-hosting the core inference workload genuinely makes economic sense. You can absorb the ops burden because you already have the infrastructure discipline. A four-GPU node handling classification and summarization workloads, with the API reserved for frontier-quality tasks, produces meaningful cost savings — often $8,000–$15,000 per month — that justify the engineering investment within a quarter.

The risk here is over-committing. Organizations in this range often decide to go "fully self-hosted" to maximize cost savings, procure an 8×H100 node, and then discover that peak load is 4–6× average load, leaving the hardware 30–40% utilized during off-hours and producing a higher per-token cost than they modeled. The mitigation is to start with demand-matching capacity, measure utilization for 60 days, and scale out only when you have evidence that the hardware will be utilized above 70%.

The second risk in this bracket is treating the decision as binary. The API is not a fallback of last resort — it is an elastic overflow valve that you should design into the architecture from the beginning. Even an organization that self-hosts the majority of its inference volume should have an API fallback path for peak overflow and for tasks that genuinely require frontier model quality. Building that dual-path architecture from the start is significantly easier than retrofitting it after you have built deep dependencies on a single endpoint.

The regulated industry

You are in financial services, healthcare, or defense. Data residency, regulatory compliance, or customer contractual requirements make API-based inference legally complicated or impossible for the data that matters most.

For this archetype, the cost calculus is secondary. The compliance requirement sets the floor: self-hosted, on compute in the right region, with a data handling architecture that satisfies your legal team. The interesting optimization is minimizing the footprint of the self-hosted cluster: not everything requires the same compliance posture. Internal tooling, engineering assistants, and non-PII workloads can often use API endpoints. Only the regulated workloads — the ones touching patient data, financial records, or classified information — need to be fully self-hosted. Segmenting by data sensitivity lets you minimize the self-hosted surface area while meeting all compliance requirements.

The enterprise platform

You are running inference at a scale where a dedicated ML platform team exists. API spend is above $100,000 per month. You have the scale to think about multi-model serving, model versioning, A/B testing between open and frontier models, and optimization strategies like speculative decoding and prompt caching.

At this scale, a full self-hosted deployment is almost certainly already happening. The interesting decisions are at the margin: which tasks continue to route to the API because they genuinely require frontier model capability, how you benchmark incoming open-weight model releases against your production quality bar before cutover, and how you structure the commercial relationship with GPU cloud providers (reserved versus on-demand, single-vendor versus multi-cloud). The decision matrix in this post applies, but you are likely operating in the upper-right quadrant of it permanently.

The fix: decision matrix, hybrid architecture, and implementation

The decision matrix

Use this table as a structured starting point. Fill in your actual values for each input dimension, then read across to the recommendation column.

The table is deliberately unambiguous on the compliance row: if you have a hard data residency or air-gapped requirement, the API column is unavailable regardless of what every other row says. Compliance is not a cost consideration. It is a constraint.

Two rows in the table deserve additional commentary because they are frequently misjudged.

Peak-to-average load ratio: This is the utilization driver that determines whether your self-hosted economics work in practice. A workload with 2× peak-to-average ratio can sustain 60–70% average utilization on right-sized hardware, producing favorable unit economics. A workload with 6× peak-to-average ratio cannot: to avoid queuing at peak, you must provision for peak capacity, which means running at 15–20% average utilization and paying full hardware cost for that idle time. Most consumer-facing products have peak-to-average ratios of 4–8×, driven by time-of-day and day-of-week patterns. Most internal tools and batch pipelines have ratios of 2–3×. Know yours before sizing hardware.

Model update cadence: This row is about operational cost, not technical capability. If your business requires the latest frontier model improvements — because your product's quality bar is directly tied to how good the current best model is — then API is structurally the right choice. Frontier model updates from Anthropic, OpenAI, and Google happen on a monthly-to-quarterly cadence and are available to API users immediately. Open-weight model releases follow a different cadence and require internal evaluation, quantization, and cutover work before they reach production. If you are running a product where "we are on last quarter's model" is acceptable, the cadence is manageable. If not, the update overhead is a real cost that must be included in the self-hosted analysis.

The hybrid routing architecture

The hybrid model requires three components: a classifier that scores request complexity, a routing layer that dispatches to the appropriate inference endpoint, and a fallback path that escalates to the API when the local model's confidence is low.

Here is a minimal implementation of the router in TypeScript, suitable for a Node.js API gateway:

// hybrid-router.ts
// Routes LLM requests between self-hosted small model and API frontier model
// based on complexity classification.
 
import Anthropic from "@anthropic-ai/sdk";
import OpenAI from "openai";
 
type ComplexityClass = "simple" | "moderate" | "complex";
type InferenceBackend = "local" | "api";
 
interface RoutingDecision {
  backend: InferenceBackend;
  model: string;
  rationale: ComplexityClass;
}
 
interface RequestFeatures {
  promptTokenCount: number;
  hasCodeGeneration: boolean;
  hasMultiStepReasoning: boolean;
  hasLongContext: boolean; // > 8k tokens
  isClassificationTask: boolean;
  isExtractionTask: boolean;
}
 
function classifyComplexity(features: RequestFeatures): ComplexityClass {
  // Simple: classification, short extraction, intent routing
  if (
    (features.isClassificationTask || features.isExtractionTask) &&
    features.promptTokenCount < 2048 &&
    !features.hasMultiStepReasoning
  ) {
    return "simple";
  }
 
  // Complex: code gen, long context, multi-step reasoning
  if (
    features.hasCodeGeneration ||
    features.hasMultiStepReasoning ||
    features.hasLongContext
  ) {
    return "complex";
  }
 
  return "moderate";
}
 
function routeRequest(features: RequestFeatures): RoutingDecision {
  const complexity = classifyComplexity(features);
 
  switch (complexity) {
    case "simple":
      return {
        backend: "local",
        model: "qwen3-7b", // served via vLLM on local endpoint
        rationale: "simple",
      };
    case "complex":
      return {
        backend: "api",
        model: "claude-sonnet-4-6", // Anthropic API
        rationale: "complex",
      };
    case "moderate":
      // Default moderate to local; escalate to API on low-confidence output
      return {
        backend: "local",
        model: "qwen3-70b",
        rationale: "moderate",
      };
  }
}
 
// Usage: extract features from incoming request, get routing decision, dispatch
async function handleRequest(
  userPrompt: string,
  systemPrompt: string
): Promise<string> {
  const features: RequestFeatures = extractFeatures(userPrompt, systemPrompt);
  const decision = routeRequest(features);
 
  if (decision.backend === "local") {
    return await callLocalEndpoint(decision.model, userPrompt, systemPrompt);
  } else {
    return await callAnthropicAPI(userPrompt, systemPrompt);
  }
}
 
function extractFeatures(
  userPrompt: string,
  systemPrompt: string
): RequestFeatures {
  const combined = systemPrompt + " " + userPrompt;
  const tokenEstimate = Math.ceil(combined.split(/\s+/).length * 1.3);
 
  return {
    promptTokenCount: tokenEstimate,
    hasCodeGeneration:
      /\b(write|implement|code|function|class|script)\b/i.test(combined),
    hasMultiStepReasoning:
      /\b(step by step|reason|analyze|evaluate|compare)\b/i.test(combined),
    hasLongContext: tokenEstimate > 8192,
    isClassificationTask:
      /\b(classify|categorize|label|tag|detect intent)\b/i.test(combined),
    isExtractionTask:
      /\b(extract|parse|pull out|identify|find all)\b/i.test(combined),
  };
}
 
async function callLocalEndpoint(
  model: string,
  userPrompt: string,
  systemPrompt: string
): Promise<string> {
  // vLLM OpenAI-compatible endpoint
  const localClient = new OpenAI({
    baseURL: process.env.VLLM_ENDPOINT ?? "http://localhost:8000/v1",
    apiKey: "not-used", // vLLM does not require auth by default
  });
 
  const response = await localClient.chat.completions.create({
    model,
    messages: [
      { role: "system", content: systemPrompt },
      { role: "user", content: userPrompt },
    ],
    temperature: 0.2,
    max_tokens: 512,
  });
 
  return response.choices[0].message.content ?? "";
}
 
async function callAnthropicAPI(
  userPrompt: string,
  systemPrompt: string
): Promise<string> {
  const client = new Anthropic();
 
  const message = await client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 2048,
    system: systemPrompt,
    messages: [{ role: "user", content: userPrompt }],
  });
 
  const block = message.content[0];
  return block.type === "text" ? block.text : "";
}

The classifier in extractFeatures is deliberately rule-based. You can replace it with a small model call — a Qwen 3-7B prompt that returns a complexity score — but rule-based classification is faster, costs nothing, and is predictable. Start with rules, upgrade to a model-based classifier only if the rule-based system is producing visibly wrong routing decisions.

The routing layer adds minimal latency — a few milliseconds for the rule-based implementation above — and is not on the critical path for either backend. The more important consideration is observability: log every routing decision, including the features that drove it, so you can audit the routing behavior in production. A misclassified request — one that should have gone to the API but was sent to the local model — will produce a quality regression that is invisible unless you are logging which backend served each request. Instrument the router from day one.

A classification router built on a small self-hosted model can be faster for complex signals. Here is an example using a Qwen 3-7B completion to score query complexity:

# complexity-classifier.py
# Uses a self-hosted Qwen3-7B as a fast complexity scorer
# Runs before the main routing decision for borderline cases
 
import httpx
import json
from enum import Enum
 
CLASSIFIER_ENDPOINT = "http://localhost:8001/v1/chat/completions"
CLASSIFIER_MODEL = "qwen3-7b"
 
CLASSIFIER_SYSTEM_PROMPT = """You are a query complexity classifier for an LLM routing system.
Given a user query, output a JSON object with exactly one field: "complexity".
Values: "simple" (classification, extraction, short Q&A), "moderate" (summarization, short reasoning),
or "complex" (multi-step reasoning, code synthesis, long document analysis).
Output only valid JSON. No explanation."""
 
async def score_complexity(query: str) -> str:
    async with httpx.AsyncClient(timeout=2.0) as client:
        try:
            response = await client.post(
                CLASSIFIER_ENDPOINT,
                json={
                    "model": CLASSIFIER_MODEL,
                    "messages": [
                        {"role": "system", "content": CLASSIFIER_SYSTEM_PROMPT},
                        {"role": "user", "content": f"Query: {query[:512]}"},  # cap at 512 chars
                    ],
                    "temperature": 0.0,
                    "max_tokens": 32,
                    "response_format": {"type": "json_object"},
                },
            )
            data = response.json()
            result = json.loads(data["choices"][0]["message"]["content"])
            return result.get("complexity", "moderate")
        except Exception:
            # Classifier failure falls through to moderate (conservative default)
            return "moderate"

The 2-second timeout is intentional and critical: if the classifier is slow or unavailable, the main request should not be blocked. The except clause catches both network failures and malformed JSON, falling back to "moderate" — a conservative default that routes to the local 70B model rather than the API. You can invert this to fall back to "complex" (routing to the API) if you prefer to err on the side of quality rather than cost.

The vLLM production setup

vLLM (docs.vllm.ai) is the de facto production inference server for open-weight models in 2026. It implements PagedAttention — the key innovation described in the Kwon et al. 2023 ArXiv paper "Efficient Memory Management for Large Language Model Serving with PagedAttention" (arxiv.org/abs/2309.17453) — which enables non-contiguous KV cache storage and dramatically improves GPU memory utilization at high concurrency.

A minimal Kubernetes deployment for vLLM serving Qwen 3-70B on 4×H100:

# vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen3-70b
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-qwen3-70b
  template:
    metadata:
      labels:
        app: vllm-qwen3-70b
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.6.2
          args:
            - "--model"
            - "Qwen/Qwen3-70B-Instruct"
            - "--tensor-parallel-size"
            - "4"
            - "--max-model-len"
            - "32768"
            - "--gpu-memory-utilization"
            - "0.92"
            - "--enable-chunked-prefill"
            - "--served-model-name"
            - "qwen3-70b"
            - "--port"
            - "8000"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 4
              memory: "256Gi"
            requests:
              nvidia.com/gpu: 4
              memory: "256Gi"
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
            - name: NCCL_DEBUG
              value: "WARN"
            - name: VLLM_WORKER_MULTIPROC_METHOD
              value: "spawn"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 30
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 90
            periodSeconds: 10
            failureThreshold: 5
      nodeSelector:
        accelerator: h100
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen3-70b
  namespace: inference
spec:
  selector:
    app: vllm-qwen3-70b
  ports:
    - port: 8000
      targetPort: 8000
  type: ClusterIP

Key configuration decisions in this manifest:

• tensor-parallel-size: 4 splits the model across four GPUs. Required for 70B at BF16 precision; it will not fit in one GPU's VRAM.
• gpu-memory-utilization: 0.92 leaves 8% of VRAM for the CUDA context and OS overhead. Setting this to 0.95 or higher frequently produces OOM on the first large request batch.
• enable-chunked-prefill improves throughput by allowing long prefill operations to be interleaved with decoding, reducing head-of-line blocking on long-prompt requests.
• max-model-len: 32768 caps the context window. Qwen 3-70B supports up to 131,072 tokens, but KV cache cost grows quadratically with context length. Set this to the longest context your actual requests use, not the model's maximum.

The Python vLLM serve invocation for direct non-Kubernetes deployment:

# serve-qwen3-70b.py
# Run with: python serve-qwen3-70b.py
# Requires: pip install vllm>=0.6.0
 
import subprocess
import os
 
VLLM_CMD = [
    "python", "-m", "vllm.entrypoints.openai.api_server",
    "--model", "Qwen/Qwen3-70B-Instruct",
    "--tensor-parallel-size", "4",
    "--max-model-len", "32768",
    "--gpu-memory-utilization", "0.92",
    "--enable-chunked-prefill",
    "--served-model-name", "qwen3-70b",
    "--host", "0.0.0.0",
    "--port", "8000",
    "--trust-remote-code",  # Required for Qwen3 architecture
    "--dtype", "bfloat16",
    "--disable-log-stats",  # Remove for debugging; adds overhead at high QPS
]
 
env = os.environ.copy()
env["NCCL_DEBUG"] = "WARN"
env["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
 
if __name__ == "__main__":
    subprocess.run(VLLM_CMD, env=env, check=True)

Monitor GPU health and utilization with:

# Real-time GPU utilization across all devices in a multi-GPU node
watch -n 2 nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu \
  --format=csv,noheader,nounits
 
# Check NCCL communication health (useful for diagnosing tensor-parallel issues)
nvidia-smi topo -m
 
# Tail vLLM request logs for throughput and queue depth
journalctl -u vllm -f | grep -E "(prompt_tokens|generation_tokens|request_id|waiting)"
 
# GPU memory profiling snapshot (run during load)
nvidia-smi --query-compute-apps=pid,used_memory --format=csv

SGLang as an alternative

SGLang (sgl-project.github.io) is the second major inference server worth evaluating. It diverges from vLLM in two meaningful ways: its RadixAttention KV-cache reuse mechanism (which provides higher cache hit rates on requests that share a common prefix, such as a fixed system prompt), and its constrained decoding engine for structured output, which outperforms vLLM's guided decoding in throughput on JSON-schema-constrained workloads.

If your workload involves a fixed system prompt (common in RAG pipelines, classification, or instruction-following setups), SGLang's prefix caching can reduce effective TTFT by 40–60% for the second-and-subsequent requests sharing that prefix. If your workload requires high-throughput structured JSON output — extraction pipelines, schema-constrained generation — SGLang's constrained decoding is faster.

The practical guidance: benchmark both on your workload. vLLM has a larger community, more extensive documentation, and broader model support. SGLang is faster in specific scenarios and is worth the evaluation if either of the above patterns applies to you.

TGI (HuggingFace Text Generation Inference) was the dominant inference server for open-weight models through 2024. In 2026 it occupies a legacy position: most organizations that deployed TGI in production are still running it because migration has a cost, not because they would choose it today. For new deployments, vLLM or SGLang is the default choice, and TGI is rarely the right answer for new infrastructure. If you are on TGI today, the migration path to vLLM is well-documented (vLLM exposes an OpenAI-compatible API, so the client-side change is minimal) and most organizations find throughput improvements of 20–40% after migration, primarily due to PagedAttention's superior KV cache management versus TGI's earlier attention allocation strategy.

One emerging pattern worth tracking: speculative decoding. Both vLLM and SGLang support speculative decoding, in which a small draft model proposes candidate token sequences that the large target model then verifies or rejects in a single forward pass. For workloads where the draft model's predictions are frequently accepted — short, predictable outputs, continuation tasks, templated generation — speculative decoding can improve effective throughput by 1.5–2.5× without degrading output quality. The operational complexity is higher (two models to manage, draft model must be loaded alongside the main model), but at scale the throughput gain is significant enough to materially reduce the hardware required per unit of output.

Capacity planning

The capacity planning failure mode is buying for average load and discovering that peak is 4–6× higher. For inference workloads, this produces either request queuing (acceptable for batch jobs, catastrophic for real-time UX) or overprovisioning (expensive and wasteful).

The correct approach is to profile your existing API workload before sizing self-hosted capacity:

1. Export one month of API call logs, including timestamps and token counts.
2. Plot requests-per-minute across the day and week. Identify P95 and P99 peaks.
3. Size self-hosted capacity for P95 peak throughput, not average.
4. Add autoscaling if your cloud GPU provider supports it (CoreWeave and Lambda Labs have autoscaling APIs for on-demand nodes).
5. For traffic above the P95 threshold, route to the API as overflow. This is not a failure — it is an intentional burst path that costs slightly more than self-hosted but far less than overprovisioning for peak.

The burst-to-API pattern is the hidden advantage of hybrid architecture: you are never fully committed to self-hosted at peak load. The API absorbs spikes. Your self-hosted cluster handles the steady-state baseline at low cost, and the variable cost of the API is only incurred for the small fraction of traffic that exceeds it.

Ops burden: the invisible line item

The operational burden of a self-hosted inference cluster is systematic enough that it is worth decomposing explicitly. These are the categories of work that must be owned, staffed, and on-called:

Inference server management: vLLM and SGLang release frequently. Version updates often include performance improvements, bug fixes, and support for new model architectures. Running a version that is more than 3–4 releases behind the current stable release means you are likely missing meaningful throughput improvements. Staying current requires scheduled maintenance windows and regression testing on your workload before cutover — call it 4–8 hours of engineering time per minor release, and 1–2 days for major version upgrades.

Model weight management: Model weights for a 70B model in BF16 are approximately 140GB. Storing multiple versions — current production, staged candidate, rollback — requires significant storage. For a 4-GPU node with 3 model versions, plan 500–800GB of fast NVMe storage on the node. Retrieval from object storage (for rollback scenarios) takes 15–30 minutes for a 70B model at 5 Gbps network throughput. This means rollback after a bad model update is not instantaneous — design your update procedure accordingly.

Failure mode ownership: The failure modes of a self-hosted GPU cluster are different from application software failures. OOM (out-of-memory) events during large batch inference do not produce a stack trace; they produce a CUDA error and a dead process. NCCL timeout during tensor-parallel communication manifests as a hung inference request that neither completes nor times out unless you have explicit request timeouts configured. Driver mismatches after a kernel update silently degrade throughput without producing an error. GPU hardware failures (which occur at a rate of approximately 0.5–2% per GPU per year at scale) require physical or cloud-provider intervention. None of these failure modes appear in your application logs. All require specialized knowledge to diagnose.

Capacity planning and autoscaling: Unlike application servers that scale on CPU/memory metrics, GPU inference clusters scale on queue depth and utilization. A vLLM endpoint that is 100% utilized does not reject requests — it queues them, and queue depth grows until TTFT becomes unacceptable. Autoscaling on GPU clusters is slower than application server autoscaling: a new GPU node takes 2–10 minutes to provision and load model weights versus 30–90 seconds for a standard application container. Build monitoring alerts on request queue depth and TTFT P95, not on CPU or memory utilization, and trigger scale-out earlier than you think you need to.

Security surface: An inference endpoint that accepts arbitrary text input requires careful thought about prompt injection, denial-of-service via long-context requests, and access control. A self-hosted vLLM endpoint running without authentication on an internal network is not protected against an internal attacker or a compromised service that can reach it. Add authentication (even a simple API key), rate limiting, and maximum input length enforcement to every production inference endpoint regardless of whether it is internally or externally exposed.

The fully-loaded ops cost for all of the above — including the on-call rotation, the model update cycles, the storage, and the engineering time to manage incidents — is not $6,700/month for a well-staffed team. That number is a floor for a mature infra team that already has robust tooling in place. For a team standing up GPU infrastructure for the first time, expect 1.0–1.5 FTE equivalent in the first six months as processes, playbooks, and monitoring are built from scratch. Budget accordingly.

GPU rental landscape

GPU rental in 2026 has matured into three tiers with distinct tradeoffs:

Reserved instances on specialized providers (CoreWeave, Crusoe Energy, Lambda Labs) offer 20–40% discounts over on-demand pricing in exchange for 1-year or 3-year commitments. CoreWeave's reserved H100 SXM5 pricing is approximately $1.80–$2.10/hr at 1-year commitment versus $2.60–$3.20/hr on-demand. Crusoe Energy (crusoecloud.com) operates on stranded energy and is competitively priced for bulk reserved capacity. These are the right choice once you have stable, predictable load and have been running self-hosted for at least 90 days.

On-demand spot markets (RunPod, vast.ai) offer lower sticker prices but with availability variability and potential interruption risk. RunPod's H100 NVL on-demand runs $2.29–$2.85/hr depending on region. vast.ai runs a marketplace model where prices are set by host operators, typically 10–30% cheaper than RunPod for equivalent hardware. These are the right choice for early experimentation, for bursty workloads, or for organizations that want maximum flexibility before committing to reserved capacity.

Hyperscaler GPU instances (AWS p4de, Azure ND H100, GCP A3 Ultra) are the most expensive option — typically 30–50% above specialized provider pricing — but they offer integrated billing, native VPC connectivity, compliance certifications that specialize providers do not have, and SLA commitments that matter in regulated industries. For organizations already deeply invested in a single hyperscaler's ecosystem, the operational simplicity of staying on one bill may outweigh the cost premium.

Hosted open-weight providers — Together AI, DeepInfra, and Fireworks AI — have emerged as a significant fourth option that the API-vs-self-hosted binary misses. These providers run Qwen 3-70B, Llama 4, and Mistral on their infrastructure and charge by the token, similar to Anthropic and OpenAI, but at prices reflecting the lower cost of open-weight models. Together AI (together.ai) charges approximately $0.90 per million tokens for Qwen 3-72B as of June 2026. DeepInfra (deepinfra.com) and Fireworks AI (fireworks.ai) are in the $0.50–$1.50 range depending on model and context length. This is a legitimate middle path: open-model quality without self-hosted ops burden, at substantially lower cost than frontier API pricing.

The hosted open-weight option is particularly compelling for organizations in the $5,000–$20,000/month API spend range who would benefit from routing easy tasks to a cheaper model endpoint but do not want to operate self-hosted infrastructure. You get the cost savings of open-weight model pricing without owning the ops. The tradeoff is data handling: data still leaves your infrastructure, so this path is not available for air-gapped or strict data residency requirements.

The total GPU cloud pricing landscape in June 2026, across tiers:

Prices are approximate and as of June 2026 from public pricing pages. Negotiate discounts for volume commitments above five nodes or for long-term contracts above two years.

The hyperscaler price premium is substantial — 40–80% over specialized providers at equivalent hardware specs. What you get for that premium: integrated billing with the rest of your cloud spend, native VPC/private networking without additional configuration, SLA commitments backed by enterprise contracts, compliance certifications (SOC 2, ISO 27001, FedRAMP) already in place, and support teams who know your account. For organizations where one or more of those items has clear monetary value — a large enterprise contract that requires hyperscaler-level compliance attestations, for instance — the premium is justified. For organizations for whom the specialized providers' compliance posture is sufficient, it usually is not.

Anti-patterns

These are the five most common ways to arrive at the wrong architecture. Each is recognizable in retrospect and preventable with the right framework upfront.

Anti-pattern 1: Moving to self-hosted purely to save money, without modeling on-call cost.

This is the most common error. The analysis stops at "GPU rental costs $X/month, API costs $Y/month, X < Y, we should self-host." The missing terms are SRE time (real, measurable, usually 0.3–0.7 FTE), model update cycles (a new Qwen release requires eval, cutover, and testing — plan 2–5 days of engineering per major version), and the cost of incidents that would not have occurred on a fully managed endpoint. When those terms are included, many organizations discover that the net savings are much smaller than projected, and sometimes negative.

Anti-pattern 2: Buying a large multi-GPU node and discovering peak-to-average ratio makes it chronically underutilized.

An 8×H100 node costs $16,000/month whether it is running at 20% or 95% utilization. If your average workload uses 2 GPUs worth of compute and your peak uses 6, you are paying for 8. The first step is profiling — export your actual token throughput by hour for a month and compute the utilization curve. Start with a 4-GPU node if the data supports it. Add a second node when you have evidence of sustained >70% utilization, not before.

Anti-pattern 3: Skipping evaluation and assuming open-model quality equals API quality.

Benchmarks are not your workload. A model that scores well on MMLU or HumanEval may perform meaningfully worse on your specific domain, prompting style, or output format requirements. Before committing to a self-hosted model in production, run a minimum of 500 representative production requests through both the API model and the candidate self-hosted model and score the outputs on your actual quality metric — not a proxy. This evaluation sprint takes one to two weeks. Not doing it and then discovering a quality regression in production costs far more.

Anti-pattern 4: Locking into a 1-year reserved GPU contract at the start.

Reserved pricing is attractive and real. It should not be taken until you have 60–90 days of production self-hosted data establishing that your utilization is high enough and stable enough to justify the commitment. Early in a self-hosted rollout, the workload, the model choice, and the capacity sizing are all provisional. A 1-year contract locks you into a specific hardware configuration during the period when you are most likely to want to change it.

Anti-pattern 5: Running Ollama in production at scale.

Ollama (ollama.ai) is an excellent tool for local development, rapid prototyping, and single-user inference on developer hardware. It was not designed for multi-user, high-concurrency production serving. It lacks continuous batching — each request is served sequentially, meaning that at N concurrent users, the N-th user waits for N-1 requests to complete before theirs begins. This is acceptable for one developer on a laptop. It is disqualifying for a production endpoint serving more than a handful of concurrent users. Use vLLM or SGLang for production.

The transition from Ollama to a production inference server is more than a config file change. It requires rethinking how model weights are loaded, how concurrency is managed, how the endpoint is monitored, and how rolling updates are handled. The earlier in the product lifecycle you make this transition, the lower the migration cost. Deploying Ollama in production with the intent of "migrating to vLLM when we need to" consistently results in the migration happening during a latency crisis — the worst possible time to do infrastructure work.

Anti-pattern 6: Self-hosting in a region without available GPU hardware.

GPU availability varies significantly by geography. EU-region GPU availability is materially tighter than US-East. Regulated organizations building for EU data residency sometimes discover that the compute they want — H100 SXM5, for example — is only available on long waiting lists from EU-region providers. Plan procurement 8–12 weeks in advance for reserved capacity in constrained regions. For on-demand spot market capacity, build availability monitoring into your deployment pipeline and have a fallback region.

Anti-pattern 7: Not instrumenting the routing layer.

The hybrid routing architecture only produces its expected cost savings if the routing decisions are correct and measurable. Organizations that build a routing layer without instrumentation discover problems late: the classifier that was working well on the original request distribution starts misrouting after a product change that shifts the query pattern, and no one notices for weeks because there is no dashboard showing the fraction of requests going to each backend or the quality differential between them. Instrument every routing decision — backend chosen, classifier output, request features, response latency — and build a dashboard that shows the routing distribution and any quality signal you have. Anomalies in the routing distribution are often the first signal that the classifier needs retraining or that the request pattern has shifted.

Anti-pattern 8: Conflating model-level and system-level latency.

The TTFT numbers quoted in benchmarks — 50ms for self-hosted, 200ms for API — are model-level numbers. System-level latency adds authentication overhead, load balancer processing, request queuing, network transit, and response serialization. In a well-architected system, these add 10–30ms. In a poorly architected system — routing through multiple proxies, synchronous auth calls, large request bodies — they can add 200–400ms and completely eliminate the TTFT advantage of self-hosted. Measure system-level latency from the client's perspective, not from the inference server's perspective, when making architecture decisions.

Case studies

These are three real-world deployment patterns, anonymized. The numbers are composites calibrated to representative ranges; they are not exact figures from a single organization.

Case study 1: the startup that moved to self-hosted and won

A Series A applied AI startup was running all inference through Anthropic's API. By Q1 2026, spend had reached $14,000/month, growing roughly 15% month-over-month as the product scaled. The founding team projected $40,000/month by year-end if growth continued.

The CTO initiated a self-hosting evaluation. After profiling two months of API logs, the team found that 68% of their token volume was flowing through a summarization and classification pipeline that ran on short (< 2,048 token) inputs and required moderate — but not frontier — quality. The remaining 32% of volume was a synthesis feature that users valued highly and where model quality was visibly important.

Architecture decision: a 4×H100 node on Lambda Labs ($8,000/month on-demand, later converted to a reserved instance at $6,200/month) running Qwen 3-70B via vLLM for the summarization and classification pipeline. The synthesis feature remained on Anthropic's API with Claude Sonnet 4.6.

All-in monthly cost after cutover:

• 4×H100 reserved: $6,200
• Anthropic API (synthesis only, 32% of original volume): $2,800
• Infra engineering allocation (0.4 FTE, split with other infra work): $4,200
• Total: $13,200/month

Net saving versus all-API baseline: approximately $800/month. Not transformative at the current volume. But token volume tripled over the following six months as the product grew — without proportionate API cost growth. At the new volume, the savings were $18,000/month on a marginal basis. The investment in self-hosted capacity paid off not because of the immediate cost reduction but because it changed the marginal cost structure at scale.

The additional benefit: TTFT on the summarization pipeline dropped from a P50 of 420ms to a P50 of 95ms. This produced a measurable improvement in the product's perceived responsiveness that the team had not modeled in advance.

Several things went wrong during the rollout that are worth noting, because they are representative of the category of problems that cost-only analyses do not surface:

The first production incident occurred on day four of the new deployment. A batch of unusually long documents — some users were pasting full-length reports rather than excerpts — generated prefill sequences exceeding 24,000 tokens, which hit the max-model-len limit and caused vLLM to return errors without a clear error message visible to the frontend. The fix was straightforward (increase max-model-len and add client-side input truncation), but the incident ran for ninety minutes before the root cause was identified, because the on-call engineer's first instinct was to check application logs rather than vLLM logs. Adding vLLM logs to the primary alerting pipeline was a post-incident action that should have been done before deployment.

The model update from Qwen 3-70B to a subsequent release three months after launch required two days of engineering work: one day for the offline evaluation against 500 representative requests, and one day for the staged rollout (10% → 50% → 100% traffic) with production monitoring. This is the model update cadence cost that the initial projections had budgeted as "a few hours." The actual number was more realistic — two days per major model update, roughly quarterly — but it was higher than the initial estimate and contributed to a perception among engineering leadership that self-hosted was more expensive than projected. Calibrating expectations on model update cost before launch reduces this kind of perception problem.

Case study 2: the fintech forced to self-host

A European fintech operating under GDPR with contractual commitments to enterprise customers that personal financial data would not leave their designated EU-region infrastructure needed to run LLM-based transaction categorization and anomaly narrative generation for a compliance reporting product.

The API option was not evaluated seriously. The legal team determined early that routing customer financial data through any third-party hosted LLM endpoint — even one with a GDPR-compliant data processing agreement — created a risk profile their enterprise customers would not accept. Self-hosting was the only option.

The deployment:

• 4×A100 80GB node in an EU-region bare-metal provider
• Qwen 3-32B (smaller model chosen for cost; the task — transaction categorization — does not require 70B capability)
• vLLM with TLS termination at the load balancer and no external network egress
• Monthly all-in cost: approximately $9,200 (hardware $6,800, networking/storage $800, ops allocation $1,600)

The cost was not the primary success criterion — the compliance requirement was. But at $9,200/month for a workload that would have cost $11,000–$15,000/month on a GDPR-BAA API endpoint at the same volume, the cost outcome was also favorable.

The key lesson: regulatory and contractual requirements are not optimized away. If the constraint is real, model the self-hosted cost as a floor and evaluate against the cost of not having the product at all, not against the API-only alternative.

The interesting secondary lesson from this case is about model selection under a compliance constraint. The temptation when forced to self-host is to run the largest available open-weight model — Qwen 3-70B — on the assumption that bigger is better. The fintech team ran an offline evaluation and found that Qwen 3-32B produced equivalent accuracy on their transaction categorization task, with a 6-point F1 difference that was below the threshold for any visible impact on the compliance reports the product generates. The smaller model required 2 GPUs (at BF16 precision) rather than 4, halved the hardware cost, and improved TTFT by 35%. The evaluation sprint cost two weeks. It saved $4,000/month for the life of the deployment.

The right model for a compliance-constrained deployment is not the largest model you can run — it is the smallest model that passes your quality bar on your actual task. The evaluation is the work. The hardware selection follows from it.

Case study 3: the SaaS that switched twice and landed on hybrid

A mid-size developer tools SaaS had an API bill of $22,000/month across code review assistance, documentation generation, and an internal search feature. They moved to a full self-hosted deployment on an 8×H100 cluster in Q4 2025 to reduce cost.

The first six months looked good. Costs dropped to $18,500/month all-in. Then two things happened. First, token volume became spiky as the product added team accounts with shared context — peak was 6× average. Second, a new frontier model release produced a material jump in code review quality that users noticed and asked for, and deploying the equivalent open-weight model required a two-week evaluation and cutover process.

By Q2 2026, the team had realized they were in the wrong architecture. The 8×H100 cluster was 35% utilized on average, 90% at peak. They were paying for peak capacity constantly. And the engineering overhead of managing open-weight model updates — Qwen 3 to Qwen 3.5 to Qwen 4 — was consuming engineering time they would rather have spent on product.

The resolution: a hybrid architecture. The documentation generation and search features moved to a self-hosted 4×A100 cluster (lower cost, sufficient for the task). Code review assistance — the high-value, quality-sensitive feature — moved back to the API. The 8×H100 cluster was released, eliminating $9,000/month of hardware cost.

Current all-in cost: $14,800/month (4×A100 self-hosted, documentation/search + API for code review). Net saving versus original API-only baseline: $7,200/month. Net saving versus the failed full-self-hosted attempt: $3,700/month plus recovered engineering capacity.

The lesson: full self-hosted is not always the endpoint to optimize for. Hybrid is not a compromise — for many production workloads, it is the correct architecture.

The case is also instructive about what the full-self-hosted experiment cost before the team pivoted. The 8×H100 cluster was deployed for eight months before the decision to release it. During those eight months, the team ran at approximately 35% average utilization, paying for 100% of the hardware. The sunk cost of those eight months at below-break-even economics is real. It did not make the pivot to hybrid wrong — the pivot was correct — but it illustrates that the decision to self-host should be made with explicit utilization targets and a review gate at 60 and 90 days. If utilization is not tracking toward 65%+ by day 90, revisit the architecture before committing to the trajectory for another year.

The final architecture after the pivot looked like this: documentation and search on 4×A100 self-hosted (Qwen 3-32B, optimized for the task); code review on Anthropic API (Claude Sonnet 4.6); embeddings on a single A10G node running a sentence-transformer model via vLLM. The embedding workload, which had been running on the API at a surprisingly high cost (embedding calls are cheap per token but high volume), moved cleanly to a single mid-tier GPU and reduced that specific cost by 85% with no quality impact. Embeddings are often the easiest first self-hosting win and a useful first step for organizations building GPU infrastructure discipline before taking on a larger LLM deployment.

Limits and caveats

Any framework built on specific pricing numbers has an expiration date. The numbers in this post reflect June 2026 pricing from the vendors cited. GPU pricing on both the provider side (Lambda, CoreWeave, Crusoe) and the hyperscaler side (AWS, Azure, GCP) has moved 20–40% year-over-year for the past three years. API pricing has trended downward as providers compete. Token costs for frontier models in mid-2025 were materially higher than today. Both trends — cheaper GPUs, cheaper API tokens — are likely to continue, but the rate and direction of each is genuinely uncertain.

The model quality gap between frontier and open-weight models is closing, but it has not closed. GPT-5, Claude Opus 4.8, and Gemini 2.5 Ultra hold a meaningful lead on complex reasoning, long-context coherence, and multimodal capability that open-weight 70B models have not matched as of June 2026. That gap will close further. When it will close completely — if it does — is not predictable with confidence. Any architecture that bets heavily on "open-weight models will be equivalent to frontier models within 12 months" is betting on a forecast, not an observed fact.

Hosted open-weight providers (Together AI, DeepInfra, Fireworks) increasingly blur the API-vs-self-hosted distinction in a way that is genuinely good for platform engineers. Running Qwen 3-70B on Together AI at $0.90/M tokens gives you open-model pricing without self-hosted ops. For organizations whose primary objection to self-hosting is the ops burden, this is often the right answer — not because it is perfectly optimal on cost or perfectly clean on compliance, but because it eliminates the operational problem without the full overhead of self-hosted infrastructure.

Emerging model architectures — Mamba-based state space models, mixture-of-experts variants, and speculative decoding pipelines — have the potential to reset the throughput math significantly. vLLM already supports speculative decoding, which can improve effective throughput 1.5–2× for latency-sensitive workloads by using a smaller draft model to propose token candidates that the large model verifies in bulk. If architectural innovations accelerate, the cost crossover points in the decision matrix above will shift. Treat the specific thresholds as current-best-estimates, not permanent facts.

The most significant unknown for the next 12 months is the trajectory of frontier model pricing. Anthropic, OpenAI, and Google have all reduced API prices substantially over the past two years as they scale inference infrastructure and optimize serving costs. If that trend continues — and there is no strong reason to believe it will stop — the cost crossover point will shift to higher API spend levels, making API-only competitive at volumes that today favor self-hosting. The decision matrix should be revisited annually at minimum, and any long-term capacity commitment (1-year GPU reservations, multi-year GPU procurement contracts) should account for the possibility that the economics shift materially within the commitment period.

Finally: the organizational capability factor. The decision matrix in this post treats infra team maturity as a binary or ordinal variable, but it is more nuanced than that. Building and operating a self-hosted inference cluster is a skill that compounds. A team that has operated a self-hosted cluster for 12 months has tooling, runbooks, monitoring, and institutional knowledge that a team starting fresh does not. The short-term economics of self-hosting look worse than the long-term economics, because the fixed cost of building capability is front-loaded. If you are evaluating self-hosting and your current API spend barely clears the break-even point, factor in that the economics improve as your team's capability matures — but be honest about whether you have the patience and budget to reach that maturity.

Sources and further reading

Pricing references (accessed June 2026)

• Anthropic API pricing: anthropic.com/pricing — Claude Sonnet 4.6, Opus 4.8, Haiku 3.5 input/output token costs.
• OpenAI API pricing: openai.com/pricing — GPT-5, GPT-4o pricing and context tiers.
• Google Gemini pricing: ai.google.dev/pricing — Gemini 2.5 Pro and Flash pricing.
• Lambda Labs GPU Cloud: lambdalabs.com/service/gpu-cloud — H100 SXM5 on-demand and reserved pricing.
• CoreWeave pricing: coreweave.com — Reserved H100 and A100 cluster pricing, EU region availability.
• RunPod pricing: runpod.io/gpu-instance/pricing — H100 and A100 on-demand GPU pricing.
• vast.ai marketplace pricing: vast.ai — Spot market GPU rates by hardware class.
• Together AI inference pricing: together.ai/pricing — Qwen 3-72B, Llama 4, and other open-weight model API pricing.
• DeepInfra pricing: deepinfra.com/pricing — Open-weight model inference by token.
• Fireworks AI pricing: fireworks.ai/pricing — Open-weight and fine-tuned model inference.

Technical references

• vLLM documentation: docs.vllm.ai — PagedAttention, continuous batching, tensor parallel, chunked prefill configuration.
• vLLM paper: Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," ArXiv 2309.17453 (2023). arxiv.org/abs/2309.17453
• SGLang documentation: sgl-project.github.io — RadixAttention, constrained decoding, structured output serving.
• Qwen 3 model card: huggingface.co/Qwen/Qwen3-70B-Instruct — Architecture, context length, benchmark performance.
• Llama 4 model card: huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct — Parameter count, architecture, HuggingFace benchmark results.

Industry analysis

• Andreessen Horowitz, "Who Owns the Generative AI Platform?" (a16z.com) — analysis of the API-vs-self-hosted cost dynamics in the LLM layer.
• Sequoia Capital, "AI's $600B Question" — GPU capacity, utilization, and the total cost of inference at scale.
• Lianmin Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs," ArXiv 2312.07104 (2023). arxiv.org/abs/2312.07104

<!-- audit-checklist
- [ ] Every pricing claim scoped to date of writing.
- [ ] All model + GPU claims verified against vendor docs or scoped as observation.
- [ ] All linked URLs return HTTP 200.
- [ ] No proprietary client name used without consent.
- [ ] No vertical industry reveal.
- [ ] Code samples compile on cited versions.
- [ ] Latency / throughput numbers sourced or scoped.
- [ ] Reading time matches body length to ±20 %.
-->