What “On-Premise” Actually Means Now
“On-premise” used to mean a server in your own basement. Large language models now offer more options, and most of the confusion starts when people conflate them. There are two basic postures and a middle way between them.
The first is the cloud AI API: you send your text to a provider like OpenAI, Anthropic, or Google, and their model sends an answer back. The provider runs it on its own infrastructure and, if it is a US provider, under US law even when the data centre is in the EU. Most people mean this by “using AI” today. It is the most capable option and by far the easiest to start with, but your data leaves your control to get there.
The second is a self-hosted open model: an open-weight model like Llama, Mistral, or Qwen, running on GPUs you govern. The GPUs sit in a rack in your own building, or you rent them as dedicated machines in a German or EU data centre under a German contract. The model weights and your data never leave infrastructure you control. Open models have closed a large part of the quality gap over the past two years, though not all of it on the very hardest reasoning tasks.
The contrast between “your building” and “someone’s cloud” matters less than who controls the machine your data is processed on, and under which legal regime. A dedicated GPU in a Frankfurt data centre, under a German processing agreement, gives you most of the sovereignty benefits of a basement server and spares you the capital expense and the hardware headaches. In practice, “on-premise-capable” usually means “runs on infrastructure you or a trusted German provider fully control”, not necessarily hardware you own. The hardware guide describes what running your own models takes technically.
Why Data Sovereignty Matters to the Mittelstand
Two reasons drive a German mid-sized company toward data sovereignty, and because they concern different problems, they lead to different answers.
The GDPR and the processor chain
Once your prompt contains personal data, such as a customer email, an HR file or a patient record, the provider that processes it becomes a processor in the GDPR sense. You then need a data-processing agreement, a lawful basis and, for a US provider, a valid transfer mechanism. All of that is doable, and I have described how to get it right in practice. But every provider you add to the chain brings another contract and another sub-processor list that you have to audit, document and explain to a data-protection officer or an auditor. Self-hosting collapses the chain, because you process the data on infrastructure you already control, under agreements you already have. That does not make you automatically “compliant”, and the work remains yours, but it removes a great many moving parts where compliance breaks unnoticed.
Trade secrets and the Geschäftsgeheimnis
For a lot of Mittelstand firms another reason outweighs the GDPR, and it does not concern personal data. It is the Geschäftsgeheimnis, which covers the CAD files, the process parameters, the pricing model, the supplier terms and the source code, in short the twenty years of know-how that make up the business. German trade-secret law only protects information you have taken “reasonable steps” to keep confidential. Routing your most valuable IP through a third-party API can undercut that protection, and some regulated or defence-adjacent supply chains forbid it by contract. For a machine-builder whose edge is process knowledge earned over years, “where does the data go?” is an existential question and not a compliance checkbox. The same applies to a clinic’s patient data or a law firm’s case files, because some information is not yours to send anywhere, however convenient the API.
The Trade-Offs
None of this makes on-premise automatically the right answer, because control costs money, capability and effort.
The capability gap. Frontier API models are still ahead on the hardest reasoning, coding, and long-context tasks. Open models are good enough for most business use cases such as drafting, extraction, classification and retrieval-augmented answers. But if your use case needs the best model on the market, you pay in accuracy for keeping data in-house.
The shape of the cost. Cloud is operating expense: you pay per token, start cheaply, and the bill grows evenly with usage. Self-hosting is capital expense or reserved GPU rental, which makes it expensive when idle and cheap at high volume. A GPU that sits idle most of the time costs many times the cloud price per request. The break-even depends almost entirely on how heavily and predictably you use it.
The operations burden. Someone has to run the system, which means patching, model updates, monitoring, GPU failures and scaling under load. A cloud API hides all of that behind a URL, and with self-hosting you take it on. No cost is underestimated more often, and it largely explains why moving from a working demo to a dependable production system is the harder half of any project.
The upside you are buying: control and auditability. On your own infrastructure you know where every byte went. You can log everything and prove data never left the country, and no vendor silently changes or deprecates the model your process depends on. In a regulated audit that proof is worth a great deal, and a black-box API cannot supply it.
Which One, and When
The decision is rarely binary, and workable architectures mostly sit in the middle.
The middle ground: EU-hosted and hybrid
Between the two extremes sit the options that most Mittelstand companies want. With EU-hosted managed inference a provider runs an open model for you inside a German or EU data centre under a proper processing agreement, so you get most of the sovereignty without owning the hardware or the on-call rota. A hybrid architecture routes by sensitivity: non-sensitive, public-facing text goes to a frontier API for maximum quality, while a self-hosted model handles anything containing personal data or trade secrets. A related pattern is to redact or tokenise the sensitive parts before anything reaches an external API. These options follow from the fact that not all of your data carries the same risk.
With these options on the table, the decision usually resolves. Lean self-hosted when the data is sensitive, meaning regulated personal data, core IP or a contractual ban on external processing, when the volume is high enough to amortise the hardware, and when open-model quality clears your bar. Stay on a cloud API when the data is not sensitive, you need the strongest possible capability, usage is spiky or low, and you do not want to carry the operations burden. Go hybrid when some of your workloads are sensitive and some are not, which describes most companies.
The choice is a business and risk decision, even though it looks technical. So do not ask which option is more secure in the abstract; ask what this data costs you if it leaks or if you lose control of it, and what the capability and operations premium of keeping it in-house buys. Your management, not just your IT, should own that question.
From our own build: keeping the decision reversible
In our own systems, the mode of operation is a setting. TippelPi, our knowledge system, runs on a Raspberry Pi with Ollama and unchanged on a GPU server with vLLM. On start-up the application checks whether a vLLM server answers and otherwise uses Ollama. The validation layer for Sentinel rules switches between cloud service and local model with two environment variables. Build this way and you can start in the cloud and move into your own network later without rewriting the application.
For the cloud variant we also measured what leaves the network. A complete run of the validation layer consists of two HTTP calls totalling 29,032 bytes: the requirement text and the column names. Log data, IP addresses and event timestamps are not among them. A measurement like this answers the question of data outflow more precisely than an architecture sketch, and it can be repeated on site. The measurement record is on the validation layer page.