Local AI server for business: what does it take?

A local AI server that a team will actually keep using starts at one GPU with 24 GB of VRAM, 64 GB of system RAM and a fast NVMe drive. That is enough to run a 27B to 32B model at good quality for a handful of concurrent users. Budget from roughly 5,000 euros for an entry machine and 12,000 to 20,000 for something a whole department leans on. The hardware is the easy part. What decides whether the project succeeds is picking the right GPU generation for the right reasons, the software around the model, an honest calculation against cloud alternatives, and having someone keep the thing running. This guide covers all of it.

What actually determines the hardware?

One number governs everything: VRAM, the memory on the graphics card. The model weights have to fit in it, along with the context window of every active conversation. System RAM, CPU and disk matter far less than people expect.

A rough rule for 4-bit quantised models, the format almost everyone runs in production:

VRAM	Model size it fits	What it feels like
8 GB	up to ~8B	Usable for simple summarisation and classification. Noticeably weaker than ChatGPT.
16 GB	up to ~14B	Decent for internal Q&A and document search.
24 GB	up to ~32B	The sweet spot. This is where local models start feeling genuinely good.
32 GB	~32B with long context	Headroom for heavy documents, still one card.
48 GB (2x24)	~32B comfortably, or 70B at 4-bit	Comfortable for a team, several users at once.
96 GB+	70B and above with room	Departmental scale, or heavy code work.

If a model does not fit in VRAM, performance collapses regardless of how fast the card is. That single fact drives every buying decision below: size by capacity first, then choose the generation.

Do we need Blackwell, the latest GPU generation?

Worth taking seriously, because the answer decides both budget and lifespan. Three generations matter in 2026:

Ampere (RTX 3090, 2020). Six years old and still everywhere in the local AI community, because used cards deliver 24 GB for 600 to 1,000 euros. The catch: they are sold as-is with no warranty, they draw more power per token than anything newer, and the silicon predates the low-precision formats modern serving is moving to.

Ada (RTX 4090, 2022). Same 24 GB ceiling, 30 to 50 percent faster, at two to three times the used-Ampere price. The awkward middle child.

Blackwell (RTX 5090 and the professional RTX 6000 line, 2025). 32 GB on the consumer flagship and up to 96 GB on a single professional board, nearly double the memory bandwidth of Ampere, native FP8 and FP4 support, and roughly 2 to 2.5 times the inference speed card for card.

For a hobby build, the used market is hard to beat: two second-hand 3090s give 48 GB for less than one new flagship, and enthusiasts squeeze remarkable results out of them. If you are a technical team happy to validate as-is hardware and live with the heat, that is a legitimate path, and we will not pretend otherwise.

For a machine a business depends on, the calculation is different, and it has little to do with keynote benchmarks. Three things dominate over a three-to-five-year horizon. Warranty and supply: a production machine needs supported, replaceable parts, and used cards are lottery tickets. Efficiency: inference cost is increasingly a power bill, and tokens per watt improved sharply across the generations; the card that is cheaper to buy is the more expensive one to run, every hour it is on. Direction of travel: open models increasingly ship in the low-precision formats that Blackwell runs natively and older silicon only emulates. Buying six-year-old architecture on a five-year depreciation plan means retiring eleven-year-old architecture.

So the honest verdict: tinkerers should buy used Ampere and enjoy it. A business deploying AI it intends to rely on should spec current-generation silicon, sized by VRAM, and treat the price difference as what it is: the cost of warranty, efficiency and staying compatible with where models are heading.

What about AMD, Intel and Apple?

Worth an honest paragraph each. AMD's 24 GB cards are attractively priced, and ROCm reached practical parity with CUDA on Linux in early 2026, but the surrounding tooling and community answers still assume NVIDIA, which matters the day something breaks. Intel's Arc Pro line delivers cheap VRAM with a young software stack. Apple's unified memory is remarkable for one person, a maxed laptop holds surprisingly large models, but memory bandwidth caps multi-user throughput, and a laptop is not a server. For a business machine serving a team in 2026, NVIDIA remains the boring, correct answer.

What is quantisation, and why does everyone use it?

Models are trained in high precision but can run in lower precision, which shrinks them dramatically. A 32B model that needs over 60 GB in full precision fits in under 20 GB at 4-bit, with a quality loss that is barely measurable for everyday work. This is not a compromise hack, it is how production local AI runs, and the newest GPUs now accelerate these formats in hardware. The practical rule: assume 4-bit, and test quality on your own documents and your own language before you commit. Benchmarks are written in English; your invoices are not.

Which models should we plan for?

The open-weight landscape moved fast, and in a good direction. In the 24 to 32B class, models like Qwen3.6 27B and Gemma 4 31B now handle general assistant work, summarisation and document Q&A at a level that would have required a data centre two years ago. Both ship under permissive licences, which matters if you plan to build on top.

Two practical notes. Licences differ more than people check: Apache 2.0 and MIT are the clean ones, while some popular families ship under bespoke community licences with usage triggers. Have someone read the licence before you standardise on a model. And for document search, retrieval quality matters more than model size. Good chunking and a solid embedding model with a 27B model will beat a 70B model with sloppy retrieval, every time.

What does the software stack look like?

This is what buyers underestimate most. A graphics card and a model file is not a product. A working setup needs five layers:

An inference engine that serves the model efficiently. vLLM is the standard for multi-user serving because it batches concurrent requests properly; Ollama and llama.cpp are excellent for single users but strain under team load.

An API layer, usually OpenAI-compatible, so internal tools can talk to your server the same way they would talk to any AI provider.

A chat interface your team actually wants to use. If it is uglier or slower than the browser tab they already know, they will go back to the browser tab, and your data walks out with them.

Retrieval, for document work. An embedding model, a vector store, chunking logic, and source citations in the answers. This is where most DIY projects stall.

Access control and logging. Who may use it, which departments see which documents, and what gets logged for audit.

None of this is exotic, all of it is open source, and wiring it together into something a non-technical team loves is a real project. Plan weeks of engineering, not days, if you build it yourself.

How many people can one machine serve?

More than most expect, because usage is bursty. People read for a minute after every answer they get.

With one 24 GB card and a 27B model, a team of 5 to 15 occasional users generally works fine. Two cards and you are comfortable at 20 to 50 for chat and document work: a 30-person firm where ten people are asking questions in any given hour barely warms the queue. Heavy continuous use, coding assistants across a whole dev team, or automated pipelines running around the clock, changes the maths completely: those workloads scale by tokens per second, not by headcount, and they are exactly where current-generation bandwidth pays for itself.

Two things make or break the experience. Proper batching in the serving layer, and keeping the model resident in VRAM at all times. A server that loads the model on every request feels broken. One that keeps it warm feels instant.

What does it cost to run?

Three numbers to plan for.

Hardware. From about 5,000 euros for a single-GPU machine, 12,000 to 20,000 for a dual-GPU workhorse on current-generation cards. Capital expenditure, depreciated over three to five years, not a subscription.

Power. A dual-GPU machine draws in the region of 700 to 900 W under load, far less when idle, and newer generations deliver meaningfully more tokens per watt. At Swedish commercial rates that lands roughly in the 400 to 900 euro per year range depending on usage.

Operations. The line item most people forget. Patching, model updates, monitoring, and fixing it when it stops. Budget a few hours of competent time per month, or buy it as a service. Ignoring it is how good hardware turns into an expensive shelf.

Is local actually cheaper than the cloud?

Sometimes. Be suspicious of anyone who says always.

Compare against the right thing. If your alternative is per-seat subscriptions, the maths favours owning surprisingly fast: enterprise AI plans typically run 40 to 60 euros per seat per month, so 30 seats is 43,000 to 65,000 euros over three years, against roughly 20,000 to 30,000 all-in for a dual-GPU machine over the same period. If your alternative is raw API usage, the API often wins on pure token economics at light and medium volume, and the break-even only arrives with heavy retrieval work, coding assistants or around-the-clock pipelines.

Scenario over 3 years	Cloud, typical range	Local, all-in range	Verdict
5 people, occasional chat	API: under 2,000	8,000 to 12,000	Cloud wins on cost. Local is a privacy decision.
30 people, chat + document search	Seats: 43,000 to 65,000	20,000 to 30,000	Owning wins, and the data stays home.
Dev team with coding assistant, heavy daily use	API: 30,000 to 90,000+	25,000 to 40,000	Owning wins with room to spare.

Ranges are deliberately wide; plug in your own numbers. One thing the table cannot show: with an enterprise cloud contract the data still leaves your network, is typically retained for abuse monitoring, and sits under foreign jurisdiction. A no-training clause does not give the data back. The honest summary: below a certain intensity you choose local for control, not savings. Above it, you get both.

How do we keep it secure?

A local server removes the third-party risk and hands you the operational one. The basics are not optional: single sign-on against your existing identity provider, so access follows employment. Network segmentation, the AI server does not need to reach the internet to serve answers, and in air-gapped setups it never does. Document permissions that mirror your file permissions, so retrieval cannot leak the salary folder to everyone. Logging that satisfies your auditors without recording every prompt forever. Encrypted disks and backups of configuration and indexes, the models themselves are replaceable.

And placement: a locked, ventilated room with stable power and wired networking. Modern GPUs under load are loud, warm and worth stealing. Physical access is access.

For fully air-gapped environments, decide up front how updates get in. That has to be a designed process with signed, verified packages, not a random USB stick someone carries in.

When is a local server the wrong choice?

Often. If your data can legally and commercially be processed by a cloud provider, and your volume is modest, the cloud is simpler and usually cheaper: we wrote a separate guide on when ChatGPT works under GDPR and what contracts it takes. If you need the absolute frontier of capability on the hardest reasoning tasks, the largest proprietary models are still ahead of what fits in a single machine. And if nobody in the organisation owns the operational side, a local server will decay quietly until people go back to pasting into a browser.

Local AI pays off when data cannot leave the building, when the volume maths above tips in your favour, when you want to stop tracking every turn in the EU-US transfer saga, or when you need to prove exactly where processing happens.

What is a sensible first step?

Not a company-wide rollout. Pick one use case with obvious value and contained scope: contract search for the legal team, a coding assistant for one dev squad, internal policy Q&A for HR. Run it for four to six weeks on real work. Measure whether people come back on their own, that is the only adoption metric that matters. Then scale to the next department with proof in hand instead of promises.

How we solve it with Blackbox

Blackbox is a machine we specify, build and deliver to your office, on current-generation hardware. Three tiers, mapping directly onto the levels in this guide: Entry with a Blackwell GPU at 24 GB VRAM and 64 GB RAM, the simplest way for a smaller team to start, and it runs models up to roughly 30B comfortably when quantised. Base, the configuration most customers choose, with two Blackwell GPUs at 48 GB combined and 128 GB RAM, built for larger models and real organisation-wide use. Enterprise on server-grade rack hardware with configurable VRAM, a Threadripper platform and 256 GB RAM and up, for deployments without a practical ceiling.

How we solve it with Blackbox

Blackbox is a machine we specify, build and deliver to your office, on current-generation hardware. Three tiers, mapping directly onto the levels in this guide: Entry with a Blackwell GPU at 24 GB VRAM and 64 GB RAM, the simplest way for a smaller team to start, running models up to roughly 30B comfortably when quantised. Base, the configuration most customers choose, with two Blackwell GPUs at 48 GB combined and 128 GB RAM, built for larger models and real organisation-wide use. Enterprise on server-grade rack hardware with configurable VRAM, a Threadripper platform and 256 GB RAM and up, for deployments without a practical ceiling.

The software stack from earlier in this guide comes assembled. Every Blackbox runs KairosOS, our software layer, with the Kairos suite on top:

Kairos Chat is the everyday AI workspace: rewrite, research, summarise and generate, in an environment you control entirely.

Kairos Index answers questions across your own documents. Every answer carries references back to the source, and the system refuses to answer when the evidence is not in your material.

Kairos Codex, now in pilot, brings AI-assisted coding into the IDE your developers already use, with source code that never leaves the network. Kairos Agents comes next.

Models are served through vLLM and stay hot in VRAM, so answers start immediately. Access control, logging and document permissions are built in, and air-gapped machines receive updates as cryptographically signed packages. We also run our own products on exactly this infrastructure, so we feel the same operational pain you would, before you do.

Typical time from order to a deployed AI layer is four to eight weeks. Read more about the software at kairos.athlas.io or the hardware at athlas.io/blackbox. We are also happy to bring a Blackbox to your office for exactly the kind of pilot described above, on your own documents. Book a demo.

Typical time from order to a deployed AI layer is four to eight weeks. You can read more about the software at kairos.athlas.io or the hardware at athlas.io/blackbox. We are also happy to bring a Blackbox to your office for exactly the kind of pilot described above, on your own documents. Book a demo.

Quick checklist before you buy

1. Define the workload first: how many people, doing what, how often.
2. Run the three-year maths against both seats and API for your volume.
3. Size by VRAM first: buy the capacity the work needs today.
4. For a production machine, prefer current-generation silicon: warranty, tokens per watt, and native support for where model formats are heading.
5. Assume 4-bit quantisation and verify quality on your own data and language.
6. Plan the full stack: serving, interface, retrieval, access control. Not just the GPU.
7. Sort placement, power, network segmentation and update process early.
8. Name the person responsible for operations, or buy that as part of the deal.
9. Start with one use case for four to six weeks, then scale on proof.

Sources

• GPU comparison for local LLM inference, 2026
• RTX 5090 vs RTX 3090 benchmarks
• Used RTX 3090 for LLMs in 2026
• Qwen model family and licences
• vLLM, high-throughput LLM serving