On-premise is back: why companies are fleeing AI cloud

After years of “everything to cloud,” organizations discover that maybe they don’t want their data traveling to someone else’s servers. Who would have thought.

---

For a decade they sold us that the future was the cloud. That having your own servers was for dinosaurs. That infinite scalability justified any cost. That worrying about where your data physically lived was paranoid.

And now, in 2026, we’re seeing the opposite movement. Companies that bet everything on cloud are bringing workloads back. Especially AI workloads.

What changed?

The three reasons for the exodus

1. The bill got out of control

Using the OpenAI API for a prototype is cheap. Using it to process millions of requests per day is another story.

I’ve seen companies that started paying €500 per month in LLM APIs and ended up with €50,000 bills. And the worst part: the cost is unpredictable. It depends on volume, prompt length, which model you use. Budgeting is nearly impossible.

With a model running on your own server, the cost is fixed. You pay for hardware (or server rental) and electricity. You know exactly what next month will cost.

2. Your data isn’t yours

When you use the OpenAI API, your prompts travel to their servers. Yes, they have privacy policies. Yes, they say they don’t train on your data (if you pay). But at the end of the day, your confidential information is on infrastructure you don’t control.

For many industries this is unacceptable. Healthcare, finance, legal, defense… they have regulatory requirements that make it impossible (or very risky) to use external clouds for sensitive data.

But even companies without strict legal requirements are reconsidering. Do you really want your company’s internal conversations, strategic documents, and customer data passing through an American company’s servers?

3. Technological sovereignty

This is the most abstract reason but perhaps the most important long-term. Depending on infrastructure from three American companies (OpenAI, Google, Microsoft) for your AI capability is a strategic risk.

What if they change prices? Change terms of service? Geopolitical restrictions? Simply decide your use case no longer interests them?

L’Oréal didn’t want to take that risk. They built L’Oréal GPT, their own internal AI platform. Not because they couldn’t afford cloud, but because they wanted control.

DeepSeek has proven you can achieve frontier-level performance with open source models. That changes the calculation completely.

The tools making it possible

Two years ago, running a decent LLM on your own server was hell. You needed expensive GPUs, deep ML knowledge, and even then performance was mediocre.

That has changed radically.

Ollama is probably the easiest way to run models locally. Install, download the model you want (Llama, Mistral, Phi, DeepSeek…), and you have a local API running in minutes. Literally minutes.

vLLM is for when you need serious performance. It optimizes inference to serve multiple requests in parallel with low latency. It’s what many companies use in production.

llama.cpp lets you run models on modest hardware, including CPUs without dedicated GPU. Performance isn’t spectacular, but it works.

LocalAI offers an OpenAI-compatible API running locally. You can migrate applications using the OpenAI API without changing code.

And open source models have improved dramatically. Llama 3, Mistral, Phi-3, DeepSeek… they’re no longer GPT-5’s “poor cousin.” For many use cases, they’re more than enough.

The realistic setup

What do you need to run AI on-premise for real?

For experimenting or personal use: A computer with 16GB RAM can run small models (7B parameters) with Ollama. Slow but functional.

For a small team: A server with an RTX 3090/4090 GPU (24GB VRAM) can serve 13-30B parameter models with decent performance. Cost: €2,000-3,000 in hardware.

For serious production: Multiple enterprise GPUs (A100, H100) or private cloud GPU services. We’re talking serious investments here, but the ROI versus APIs can be brutal if you have volume.

The hybrid option: Many companies are opting for a mixed model. Sensitive data and predictable workloads on-premise. Demand spikes and experimentation in cloud. Best of both worlds.

What nobody tells you

Running AI on-premise isn’t free in effort. You need:

• Someone to maintain the infrastructure
• Model and security updates
• Performance monitoring
• Capacity management

If you’re a 5-person startup, cloud is probably still the best option. The opportunity cost of setting up and maintaining infrastructure is too high.

But if you have an IT team, predictable volume, and data you don’t want leaving your network… the calculation has changed.

If you’re evaluating this decision for your company, I recommend first reading what nobody tells you about implementing AI in small business. Hidden costs apply here too.

My experience

I’ve been running Ollama on a VPS for months for personal tasks and experiments. The cost is fixed (what I pay for the server), latency is good, and I have total control.

For NeuralFlow, I use a combination: Claude Opus 4.5 and GPT-5 for complex tasks where performance matters, local models for batch processing and experimentation.

It’s not dogma. It’s pragmatism. Each tool for what it does best. I’m not a fanboy of any model.

The hybrid future

I don’t think we’ll return to a 100% on-premise world. Cloud has real advantages: scalability, delegated maintenance, access to frontier models you can’t run locally.

But the pendulum is swinging. After years of “everything to cloud,” companies are rediscovering the value of controlling their infrastructure.

The right answer, as almost always, is in the middle. Knowing what to put where, and why, is the skill that will define good AI architects in the coming years.

---

Have you set up on-premise AI infrastructure? What tools do you use? Share your experience.