In a single year, the share of enterprises using public cloud as the primary home for production AI inference fell from 56% to 41%. That is Broadcom's Private Cloud Outlook 2026 talking, not a hardware vendor's wishful thinking, and it describes the quietest big shift in enterprise IT: after two years of sending every prompt to somebody else's API, companies are pulling AI back inside buildings they control. The reasons are less ideological than the discourse suggests. Latency, compliance, per-token bills that scale with success, and 77% of organizations now factoring an AI vendor's country of origin into selection decisions, per Deloitte's 3,235-respondent survey.
This guide is the practical version of that shift. What “private AI” actually means (it is a spectrum, not a switch), the honest three-way cost comparison most articles collapse into a false binary, where the break-even really sits and why published answers disagree by 50x, the hardware stack at three budget tiers, and the deployment models from a private cloud to a full air gap. We build and host AI servers for a living, so the bias is disclosed up front, and so is the section on when you should not buy anything from anyone.
This guide is the practical version of that shift. What “private AI” actually means (it is a spectrum, not a switch), the honest three-way cost comparison most articles collapse into a false binary, where the break-even really sits and why published answers disagree by 50x, the hardware stack at three budget tiers, and the deployment models from a private cloud to a full air gap. We build and host AI servers for a living, so the bias is disclosed up front, and so is the section on when you should not buy anything from anyone.
The short answer
- Private AI means running open-weight models (Llama, DeepSeek, Qwen, gpt-oss) on infrastructure you control: a dedicated private cloud, your own colocated servers, on-site hardware, or an air-gapped enclave.
- The cost comparison has three rungs, not two: frontier APIs (~$7.50 to 15 per million tokens), open-model APIs (~$0.09 to 0.60), and self-hosted (~$0.30 to 0.60 at healthy utilization). Skipping the middle rung is the most common six-figure mistake.
- Break-even vs frontier APIs sits anywhere from ~5M to ~250M tokens per month depending on what you count; against open-model APIs it climbs past a billion. Compliance requirements override the math entirely.
- The stack has three tiers: a 48 to 96 GB workstation for team-scale AI, an 8-GPU server for department scale, and racks with colocation for company-wide platforms.
Why are enterprises pulling AI back in-house?
Four forces drive the repatriation, and each carries a named number.
Sovereignty: every prompt sent to an external API is company data transiting a third party, and for source code, patient records, deal documents, and anything under HIPAA, ITAR, or strict IP policy, that transit is the whole problem; 52% of surveyed enterprises cite exactly this.
Latency: 55% cite public-cloud inference latency, which matters once AI sits inside products rather than chat windows.
Cost shape: per-token API pricing is a tax on success, and IDC expects Global 1000 firms to underestimate AI infrastructure costs by 30% through 2027, the polite phrasing for bill shock.
Geopolitics: 77% of organizations now weigh an AI vendor's country of origin. None of this required open models to beat frontier ones; it required them to be good enough, which the current open-weight generation decisively is for most enterprise workloads.
Sovereignty: every prompt sent to an external API is company data transiting a third party, and for source code, patient records, deal documents, and anything under HIPAA, ITAR, or strict IP policy, that transit is the whole problem; 52% of surveyed enterprises cite exactly this.
Latency: 55% cite public-cloud inference latency, which matters once AI sits inside products rather than chat windows.
Cost shape: per-token API pricing is a tax on success, and IDC expects Global 1000 firms to underestimate AI infrastructure costs by 30% through 2027, the polite phrasing for bill shock.
Geopolitics: 77% of organizations now weigh an AI vendor's country of origin. None of this required open models to beat frontier ones; it required them to be good enough, which the current open-weight generation decisively is for most enterprise workloads.
What counts as private AI? The four deployment models
“Private” is a spectrum of control, and pretending it is binary produces bad architecture. Step one, private cloud or VPC: dedicated instances isolated by contract; the fastest start and a legitimate answer for many compliance regimes, though the machines still live in someone else's building. Step two, owned and colocated: your servers in a professional facility with contracted power, cooling, and uptime, data at rest and in flight on hardware you own, without hiring a facilities team; this is where most enterprises land, and the economics follow the ownership math from our cloud GPU pricing guide. Step three, on-site: full physical control, and full responsibility for power, cooling, and failures at 3 a.m. Step four, air-gapped: no external network at all, model updates by physical media, telemetry impossible by construction; the standard for defense and the strictest regulated work, and the reason “air-gapped AI” has become its own procurement category. Climb only as high as your compliance actually requires; every step up trades convenience for control.
The three-rung cost ladder (and the false binary)
Nearly every “API vs self-hosting” analysis compares a frontier API against owned hardware and declares a winner. That comparison skips the rung where much of the market actually lives.
Rung one, frontier APIs: GPT-5-class and Claude-class models at roughly $7.50 to 15 per million tokens blended; the best raw capability, zero infrastructure, and your data in transit to a third party on every call. Rung two, open-model APIs: the same Llama, DeepSeek, and Qwen weights you could self-host, served by providers at $0.09 to 0.60 per million; enormous savings with model freedom, but residency unchanged, the data still leaves. Rung three, self-hosting: those open weights on your own machines at roughly $0.30 to 0.60 per million at healthy utilization, with full residency, deterministic latency, and fixed costs, plus the ops burden and the utilization risk. The honest sequence for a cost-driven team is one rung at a time: if rung two solves your bill, rung three must be justified by control, not by price. If compliance forbids data leaving, rungs one and two were never options, and the ladder collapses to a hardware decision.
Where does self-hosting actually break even?
Published 2026 break-even claims span 5 million to 3 billion tokens per month, and the spread is not sloppiness; it is three different questions wearing one name. Against frontier API pricing, GPU-only math puts the crossover as low as single-digit millions of tokens per month, while fully loaded analyses that count operations and engineering put it at 100 to 256 million; the truth for your team depends on which costs are real for you, and the uncomfortable line item is people, since raw GPUs are only 30 to 40% of true self-hosting cost and a production deployment needs 1.5 to 2 engineering FTEs minimum. Against open-model API providers running optimized infrastructure at thin margins, cost-only break-even climbs past a billion tokens per month, a bar few workloads clear, which is exactly why control and compliance, not price, are the honest reasons for rung three. And when regulation decides, the math is moot: for HIPAA workloads, defense contracts, and strict IP regimes, private inference is a requirement wearing a cost costume. Compute your own crossover with your token volumes and your electricity and hosting numbers, stress-test it at 2 to 3x growth, and treat every published break-even, including ours, as a method rather than an answer.
Ready to Start Mining?
Free worldwide DDP shipping. Professional hosting from $0.055/kWh.
The private AI hardware stack, tier by tier
The stack question reduces to VRAM and concurrency, and three tiers cover 90% of enterprises. The workstation tier ($5K to 15K class): a 48 GB RTX 4090 or 96 GB RTX Pro 6000 runs Llama 3.3 70B at Q4 or Llama 4 Scout on one card, per the fits in our VRAM reference table, enough for team assistants, RAG over internal documents, and departmental copilots, plugged into an office wall socket. The server tier ($60K to 300K class): an 8-GPU machine in A100, H100, or H200 class serves 70B models at FP8 with real concurrency, runs gpt-oss 120B, and handles fine-tuning, the workhorse of department-scale private AI.
The rack tier ($500K+ class): multiple NVLinked servers for frontier-class MoE models and high-availability company-wide platforms, which is where colocation stops being optional and starts being the architecture. Which specific cards win each tier on delivered throughput is what the benchmark tool ranks; the honest sizing rule is to buy the tier your eighteen-month roadmap needs, not the one your pilot needs.
The rack tier ($500K+ class): multiple NVLinked servers for frontier-class MoE models and high-availability company-wide platforms, which is where colocation stops being optional and starts being the architecture. Which specific cards win each tier on delivered throughput is what the benchmark tool ranks; the honest sizing rule is to buy the tier your eighteen-month roadmap needs, not the one your pilot needs.
Models and software: the stack is mature now
The software objection to private AI expired quietly. On models, the current open-weight generation, Llama 3.3 and 4, DeepSeek's R1 family, Qwen3, and gpt-oss, closed the quality gap for the large majority of enterprise tasks, and our local LLM rankings track which fits which job. On serving, vLLM has become the de facto enterprise inference engine, with continuous batching and KV-cache management that turn one server into a department's worth of concurrent capacity, and the surrounding pieces (embedding models for RAG, fine-tuning via QLoRA on the training-tier hardware, observability, guardrails) are all deployable inside the perimeter. The remaining honest gap is the frontier edge: if a workload genuinely needs the absolute best model available this quarter, that model is proprietary and API-only, and the right architecture is often hybrid, private for the 90% of volume that is routine and sensitive, frontier API for the sliver that is neither.
When you should not go private
The disclosure again: we sell every layer of this stack, hardware, hosting, and GPU cloud, so weigh the following as the argument against our own interest. Do not go private below meaningful volume with no compliance driver: under a few hundred thousand tokens a day, APIs are cheaper and simpler, full stop. Do not go private without the people: the model is free, the 1.5 to 2 FTEs who keep it in production are not, and engineering time regularly exceeds infrastructure cost. Do not benchmark against the wrong rung: if an open-model API solves your bill and your data may legally transit, rung two wins and no hardware should be purchased. And do not air-gap for aesthetics: every step up the control spectrum costs real operational speed, so climb exactly as high as your regulator, your counsel, or your threat model requires, and no higher. Private AI is the right answer to specific questions, volume, residency, latency, and sovereignty, not an identity.
The bottom line
Private AI in 2026 is what the repatriation numbers say it is: a measured, accelerating pullback of inference into infrastructure enterprises control, driven by sovereignty, latency, and bills that scale with success. The playbook that survives contact with reality: map your compliance constraints first, because they may make the decision for you; price all three rungs, not two; compute break-even from your own token volumes with people costs included; size hardware to the eighteen-month roadmap using the VRAM math; and default to owned-plus-colocated at the point where the rent-vs-buy crossover says ownership wins.
The machines are in the shop, the racks are in our facilities, and the math is in this article; run it honestly and it will tell you which of the three you actually need.
The machines are in the shop, the racks are in our facilities, and the math is in this article; run it honestly and it will tell you which of the three you actually need.
Frequently asked questions
What is private AI?
Running AI models, typically open-weight LLMs like Llama, DeepSeek, Qwen, or gpt-oss, on infrastructure you control rather than through a third-party API. It spans a spectrum: dedicated private-cloud instances, your own hardware colocated in a hosted facility, on-site servers, and fully air-gapped deployments. The common thread is that your prompts, documents, and outputs stay on systems under your ownership or exclusive contract.
What is the difference between private AI and on-premise AI?
On-premise AI is the strict subset of private AI where the hardware physically sits in your building. Private AI also includes dedicated private-cloud instances and owned hardware colocated in professional facilities. The distinction matters because most enterprises get the control they actually need at the colocation step, ownership economics and data residency on their own machines, without operating power, cooling, and uptime themselves.
Is self-hosting an LLM cheaper than using an API?
Only above meaningful volume, and it depends which API. Against frontier models (~$7.50 to 15 per million tokens), published break-evens range from ~5 million to ~250 million tokens per month depending on whether engineering costs are counted. Against open-model API providers (~$0.09 to 0.60 per million), cost-only break-even climbs past a billion tokens monthly. Below a few hundred thousand tokens a day with no compliance driver, APIs win.
What hardware do you need to run an LLM privately?
Three tiers cover most enterprises. Team scale: a workstation with a 48 GB RTX 4090 or 96 GB RTX Pro 6000 runs Llama 3.3 70B quantized or Llama 4 Scout on a single card. Department scale: an 8-GPU server in A100/H100/H200 class serves 70B models at FP8 with real concurrency. Company scale: multiple NVLinked servers, usually colocated. The deciding constraint is VRAM (roughly 0.57 bytes per parameter at Q4 plus cache), then concurrency.
Which LLMs can enterprises run privately?
The current open-weight generation: Meta's Llama 3.3 70B and Llama 4 family, DeepSeek's R1 models and distills, Alibaba's Qwen3 line, OpenAI's gpt-oss 20B and 120B, plus strong smaller models (Gemma, Mistral) for lighter tasks. Licenses for these permit commercial self-hosting (check each model's terms for scale clauses). For most enterprise workloads, RAG, summarization, coding assistance, and internal copilots, the quality gap to frontier APIs has effectively closed.
What is air-gapped AI?
AI deployed on infrastructure with no external network connection at all: model weights arrive by physical media, updates are manual, and telemetry is impossible by construction. It is the standard for defense contractors, classified environments, and the strictest regulated workloads. Air-gapping costs real operational speed (no automatic updates, no external tooling), so it should be driven by an actual mandate rather than preference; most compliance regimes are satisfied at the colocation or on-premise level.
Is private AI secure and compliant for HIPAA or regulated industries?
Private deployment is how many regulated organizations satisfy requirements that forbid sensitive data transiting third parties: patient records under HIPAA, defense work under ITAR, and material non-public information in finance. The architecture keeps data on controlled systems end to end. Compliance still requires the surrounding program, access controls, audit logging, encryption at rest, and vendor agreements for colocated hardware, but private inference removes the hardest problem: the data never leaves.
Why are companies moving away from cloud AI?
Four measured reasons: data sovereignty (52% of surveyed enterprises cite keeping training data on-premises for security or compliance), latency (55% cite public-cloud inference latency), cost shape (per-token pricing scales with success, and IDC expects Global 1000 firms to underestimate AI infrastructure costs by 30% through 2027), and geopolitics (77% weigh an AI vendor's country of origin per Deloitte). Broadcom's 2026 survey shows public cloud as the primary AI inference environment falling from 56% to 41% in one year.
Should a small business self-host AI?
Usually not for cost reasons: below a few hundred thousand tokens per day, APIs are cheaper once engineering time is counted, and a production self-hosted deployment realistically needs one to two engineers' ongoing attention. The exceptions are compliance (client confidentiality, health data, IP that cannot transit third parties) and products where AI latency is the product. A middle path that works: a single 48 to 96 GB workstation running a 70B-class model for the sensitive workflows, APIs for everything else.
What is the best way to start with private AI?
Sequence it: (1) map compliance constraints, they may decide everything; (2) measure your real token volumes at production scale, not pilot scale; (3) price all three rungs, frontier API, open-model API, and self-hosted, at those volumes with people costs included; (4) if private wins, pilot on a single workstation-tier machine with your actual workload before committing to servers; (5) scale to the 8-GPU tier with colocation once utilization proves out. Buying the rack before measuring the volume is the expensive version of enthusiasm.
Sources and notes
Survey figures: Broadcom, Private Cloud Outlook 2026 (public cloud as primary AI inference environment, 56% to 41%); Deloitte (77% weigh vendor country of origin, n=3,235); Cloudian Enterprise AI Infrastructure Survey, March 2026 (55% latency, 52% data security; vendor-commissioned, treated as directional); IDC (30% cost underestimation through 2027); Gartner ($2.5T AI spending 2026, $401B infrastructure). Cost figures: frontier and open-model API ranges from published provider pricing; self-hosted per-token and break-even bands compiled from published 2026 TCO analyses, which legitimately disagree (5M to 3B tokens/month) based on comparison target, utilization, and whether operations and engineering are counted; GPUs as 30 to 40% of true self-hosting cost and 1.5 to 2 FTE minimums from enterprise deployment reports. Hardware fits follow our VRAM reference table. We sell hardware, hosting, and GPU cloud; the bias is disclosed in the article and the argument against purchase is included. All figures are mid-2026 snapshots. Hero: NASA, public domain; body: Jemimus, CC BY 2.0, via Wikimedia Commons. Diagrams: original MillionMiner graphics, free to reuse with attribution and a link to this page. Informational content, not legal or financial advice
Sources and notes
Survey figures: Broadcom, Private Cloud Outlook 2026 (public cloud as primary AI inference environment, 56% to 41%); Deloitte (77% weigh vendor country of origin, n=3,235); Cloudian Enterprise AI Infrastructure Survey, March 2026 (55% latency, 52% data security; vendor-commissioned, treated as directional); IDC (30% cost underestimation through 2027); Gartner ($2.5T AI spending 2026, $401B infrastructure). Cost figures: frontier and open-model API ranges from published provider pricing; self-hosted per-token and break-even bands compiled from published 2026 TCO analyses, which legitimately disagree (5M to 3B tokens/month) based on comparison target, utilization, and whether operations and engineering are counted; GPUs as 30 to 40% of true self-hosting cost and 1.5 to 2 FTE minimums from enterprise deployment reports. Hardware fits follow our VRAM reference table. We sell hardware, hosting, and GPU cloud; the bias is disclosed in the article and the argument against purchase is included. All figures are mid-2026 snapshots. Hero: NASA, public domain; body: Jemimus, CC BY 2.0, via Wikimedia Commons. Diagrams: original MillionMiner graphics, free to reuse with attribution and a link to this page. Informational content, not legal or financial advice
