The $2,000 Question: Should You Self-Host Your LLM?

The most common question we get: "Should we self-host our LLM or just use an API?"

The honest answer is: it depends on three numbers, and most teams can't answer all three. Here's how to work through it.

The three numbers that matter

1. Your data sensitivity score. This is the one that usually decides it. If your data falls into any of these categories, the answer is self-host, full stop:

• Client PII under legal or contractual obligations (lawyers, accountants, real estate)
• Financial data under regulatory scrutiny (firms, lenders, insurers)
• Healthcare PHI (providers, administrators, billing)
• Trade secrets or proprietary business logic (manufacturing, R&D, pricing)

For these, "we'd just send it to the API" isn't a technical choice, it's a compliance decision your legal team will eventually have to reverse.

2. Your volume. Self-hosting has a fixed cost (hardware, setup, maintenance) and near-zero marginal cost. APIs have near-zero fixed cost and per-token marginal cost. The crossover point depends entirely on how many tokens you burn per day.

Rough math: if you're processing more than a few thousand tokens per day consistently, the hardware starts making sense. If you're doing a few hundred tokens per day, the API is cheaper and you don't have to manage a GPU.

3. Your latency and uptime requirements. Self-hosted means you own the infrastructure. That's great for latency (no network hop) but it means you're on the hook for 2am failures, GPU driver updates, and model weight corruption. If you need 99.9% uptime and don't have someone who wants to be on-call for that, the API's reliability is worth the per-token premium.

The hybrid answer

Most businesses don't need an either/or. The pattern that works best:

• Sensitive data → local model (Ollama or vLLM on your hardware)
• Non-sensitive, high-volume, latency-critical → API (cheaper, more reliable, no maintenance)
• The harness → model-agnostic, so you can route each task to the right backend without rewriting agent code

That's the whole point of a model-agnostic harness. The model is a routing decision, not an architectural one. You can start with the API, add a local model when the data sensitivity demands it, and switch workloads between them without touching a single line of agent code.

The hardware question

If you're going self-host, what do you actually need?

• Small models (7B-14B): A single 24GB GPU (RTX 3090/4090) or a Mac with 32GB+ unified memory. Fine for many business use cases.
• Medium models (30B-70B): Multi-GPU or a single high-memory workstation. vLLM handles the serving.
• Large models (70B+ quantized): Getting into dedicated server territory. Usually not necessary for business agents — a well-prompted 14B model beats a poorly-prompted 70B.

The counterintuitive finding from production: for most business agent tasks (document processing, routing, structured output, tool calling), a 14B local model with a good harness outperforms a frontier API model with a weak harness. The harness is the multiplier.

The bottom line

Don't ask "should we self-host?" Ask "which of our workloads can't leave the building?" That answer determines the architecture. The rest is volume and latency, which are optimization problems, not architectural ones.

If you want us to work through your specific numbers, that's what the consulting engagement is for.