Every few weeks, someone asks us to help them stand up a private LLM. The reasoning is usually some combination of data sensitivity, regulatory pressure, and a vague sense that hosting your own model is the 'serious' choice. Sometimes they're right. Most of the time, they're not. And the cost of being wrong is significant — we're talking six to seven figures in infrastructure that sits underutilized.

The honest answer is that on-prem AI makes sense for a narrow set of use cases. The problem is that the vendors selling you GPU clusters won't tell you that, and the cloud providers pushing their API products won't tell you when you actually do need to go private. So here's how we think about it.

Start With Data Classification, Not Technology

Before you evaluate a single platform, you need to classify your data. Not in the abstract, hand-wavy way most companies do it, but with concrete tiers tied to regulatory obligations.

Tier 1 — Public or anonymized data. Marketing content, public filings, anonymized analytics. There is zero reason to run a private model for this. Use a cloud API, save yourself the headache.

Tier 2 — Internal but non-regulated data. Internal process documents, general business communications, operational data without PII. Cloud AI with proper contracts (BAAs, DPAs, data residency clauses) handles this fine. Most enterprise AI use cases live here.

Tier 3 — Regulated or sensitive data. Patient records, financial transaction data, classified operational data, anything subject to GDPR Article 9, HIPAA, or sector-specific rules like ITAR. This is where the conversation gets real. Even here, cloud options with dedicated tenancy and encryption exist — but your compliance team needs to sign off, and many won't.

Tier 4 — Sovereign or air-gapped data. Defense, certain government applications, critical infrastructure control systems. On-prem is non-negotiable. You already know this.

The Cost Comparison Nobody Wants to Do

We've run the numbers on enough deployments to give you a realistic picture. A private LLM deployment — even a modest one running a 70B parameter model — requires a minimum of $200K in GPU hardware, plus storage, networking, cooling, and the engineering team to keep it running. Annual operating costs (power, maintenance, staff) add another $150K-$300K.

Compare that to cloud API costs for the same workload. For most enterprises processing fewer than 10 million tokens per day, you're looking at $3K-$15K per month depending on the model. That's roughly $36K-$180K per year, with no capital expenditure, no hiring, and no hardware refresh cycle.

The math only flips in favor of on-prem when you're processing very high volumes (50M+ tokens/day consistently) or when regulatory requirements literally prohibit data from leaving your perimeter. For everyone else, you're paying a premium for the feeling of control.

When On-Prem Actually Makes Sense

We're not anti on-prem. We've built private deployments for clients where it was clearly the right call. Here are the scenarios where we'd push you in that direction:

You process Tier 3-4 data at scale and your regulator has explicitly ruled out cloud processing. Not 'we think they might object' — actually ruled it out. We've seen too many companies over-interpret regulatory guidance and build infrastructure they didn't need.

You need deterministic latency for real-time inference in safety-critical systems. If you're running AI in an operational loop where network latency or cloud outages could cause physical harm or massive financial loss, local inference is justified.

You have an existing GPU-capable data center and a team that can operate it. The marginal cost of adding AI workloads to existing infrastructure is dramatically lower than building from scratch. If you already have the facility and the people, the calculus changes.

The Hybrid Path Most Companies Should Take

For most regulated firms we work with, the answer is neither pure cloud nor pure on-prem. It's a tiered architecture where data classification drives deployment decisions.

Non-sensitive workloads — summarization, content generation, internal search — go through cloud APIs with appropriate contractual protections. Sensitive workloads get routed to a smaller, purpose-built on-prem model fine-tuned on your specific domain. You don't need GPT-4-class capabilities for most internal tasks. A well-tuned 7B-13B parameter model running on a single node can handle document classification, extraction, and domain-specific Q&A better than a general-purpose giant model.

This approach lets you spend your infrastructure budget where it actually matters and avoid the trap of maintaining expensive hardware for workloads that don't require it.

How to Decide

If you're weighing this decision right now, here's the framework in plain terms. First, classify your data honestly — not aspirationally. Second, check whether your regulator has actually prohibited cloud AI processing, or whether your legal team is being conservative by default. Third, run the total cost of ownership over three years, including staffing. Fourth, ask yourself whether your organization can realistically operate ML infrastructure — because buying GPUs is the easy part.

If after all that you still need on-prem, we can help you build it right. But nine times out of ten, the answer is a well-architected cloud deployment with strong governance. That's not the exciting answer. It's the honest one.