For a self-hosted LLM, GPU memory is the number that decides everything else. The rule of thumb: a model with X billion parameters needs about 2 × X GB of VRAM in FP16, far less when quantised to 4 bits, and long contexts add the KV cache on top. A small team can start with a single workstation GPU or a Mac Studio. You only need a GPU server once many people send requests at the same time.
Why you’d run a model in-house at all is covered in a separate post. This one is about the practical question that follows: what has to sit under the desk or in the rack for the thing to actually run?
How do you calculate the memory requirement?
The maths is more boring than the forum debates suggest. Hugging Face puts it like this in the Transformers docs: the weights of a model with X billion parameters need roughly 2 × X GB of VRAM in bfloat16/float16 (Hugging Face). A 7B model lands at about 14 GB, a 32B model at about 64 GB.
Quantisation brings that down a lot. In the same document a 15B model (OctoCoder) runs in 4-bit precision with “just a bit over 9GB” (Hugging Face). For everyday chat work, 4 bits is usually the sensible trade-off. You lose a little quality, and in exchange the model fits on hardware you can actually afford.
The item almost everyone forgets is the KV cache. It grows with the length of the input and the chat history. In the Hugging Face long-context example it takes about half as much memory as the model weights themselves (Hugging Face). If you plan to paste whole contracts or long manuals into the context, size generously (and test with real documents, not with “hi, who are you?”).
What are the hardware options?
Roughly, there are three routes. I’m deliberately leaving out prices; they change faster than this post will age.
| Workstation with one GPU | Mac Studio | Server with data-centre GPUs | |
|---|---|---|---|
| Memory for the model | the card’s VRAM, usually the bottleneck | shared unified memory, up to 512 GB | VRAM across several cards |
| Good fit for | a small-to-mid-size model, one user or a small team | larger models, few concurrent users | many parallel users, production applications |
| Typical stack | Ollama | Ollama | vLLM |
| Weak spot | the card’s memory ceiling | throughput under many parallel requests | purchase, power, operations |
The Mac Studio is the outsider that fits surprisingly often. Apple Silicon shares memory between CPU and GPU, and the current Mac Studio can be configured with up to 128 GB on the M5 Max and up to 512 GB on the M5 Ultra (Apple). That runs models which would otherwise need several graphics cards. Under heavy concurrent load, though, it doesn’t keep up with real server GPUs.
For a sense of the upper end: OpenAI says gpt-oss-120b runs on a single 80 GB GPU, while gpt-oss-20b runs within 16 GB of memory (Hugging Face model card).
Ollama, vLLM, Open WebUI: which stack?
Ollama is the starting point. It’s MIT-licensed (GitHub), and the workflow is short: pull a model, run it, done. One thing that matters later is that Ollama speaks a subset of the OpenAI API (Ollama docs), so applications you build against it today don’t need a rewrite when you switch.
vLLM is the choice once the experiment turns into a company-wide service. It’s built for throughput, manages the KV cache with PagedAttention, ships its own OpenAI-compatible server and is licensed under Apache 2.0 (GitHub). For a single user it’s overkill. With twenty people typing at once, you notice the difference.
Open WebUI is the chat front end most people put in front of either one: knowledge bases with document upload, roles, groups and per-model permissions (Open WebUI docs). There’s a catch. Since version 0.6.6 the project uses its own “Open WebUI License” with a branding clause. You may only remove the name and logo if no more than 50 end users access it within a rolling 30 days, or if you have written permission or an enterprise licence (Open WebUI licence). For most small businesses that’s irrelevant. It is no longer open source in the OSI sense, though, and you should know that before building it into a client project.
Which models are you allowed to use?
The licence question gets skipped because it’s “just an internal test”. Mistral Small 3.1 (model card), Qwen3-32B (model card) and gpt-oss (model card) are Apache 2.0, which makes commercial use straightforward.
Llama is different. For the multimodal models in Llama 3.2 and Llama 4, Meta does not grant usage rights to individuals living in the EU or to companies with their principal place of business there (Meta, Llama 4 use policy). Loading a Llama vision model onto the company server “just to try it” gives a European business a licence problem before the first request. Data protection is a separate matter: a local model still processes personal data, and GDPR compliance for AI stays a process, not a checkbox.
Start small, then measure
My recommendation is unexciting. Don’t buy the big box first. Take a machine you already have or can get cheaply, install Ollama and Open WebUI, load an Apache-licensed model at 4 bits and let a few colleagues do real work with it for two weeks. Afterwards you know which model size gives the quality you need, how long the contexts really get and how many requests arrive in parallel. The hardware follows almost directly from those three numbers.
If the model then runs permanently, it needs what every other service needs: updates, monitoring, backups of the knowledge base. Your IT team can do that, or you hand it over as managed hosting and operations. How it fits into a wider AI setup is on the local AI page.
Want to know which hardware is enough for your use case before you order? Get in touch — we’ll size it against your real documents.