Private AI for business: ban it, buy a GPU, or neither?
The usual answer to "our team pastes client data into ChatGPT" is either a strict ban or an expensive GPU server. I think both are wrong. Here is what private AI for business really involves, and when it is worth it.
The problem: client data in a public chatbot
Ask any small team and someone has pasted a contract, a client email or a spreadsheet into a public chatbot to "just summarise it". Nobody meant harm. It was faster. But that data now sits on someone else's servers under someone else's terms, and the business never decided that was fine.
Two popular fixes, and why I don't like them
"Ban AI." It doesn't work. People keep using it on their phones, just without telling anyone. You lose the visibility you had and keep the risk.
"Buy a big GPU server and run everything locally." It sounds like the safe choice, and sometimes it is. But I often see it sold before anyone has measured what the team actually asks. The result is an expensive machine running a model that is weaker than what people had before, so they quietly go back to the public chatbot.
The better answer is simpler: keep the documents, the history and the rules on a machine you control, and decide carefully what is allowed to go out. That is the useful core of private AI for business.
What "private" actually means
People use the word loosely, so here is how I split it:
- Fully local: a local LLM runs on your own hardware, with tools like Ollama and Open WebUI. Nothing leaves the building. This is the strictest option.
- Self-hosted, API model: the app, the documents and the history live on your server; only the question (ideally with names removed) goes to a model provider under a business agreement. Cheaper, and the answers are often better.
- Public chatbot: everything goes to the provider. Fine for general questions, not for client data.
Most small businesses I talk to don't need the strictest option for everything. They need a clear line: which data may go out, and which never does.
Where it shines: document search
The use that wins people over is almost always document search. You point a private ChatGPT-style assistant at your contracts, policies, manuals and past proposals, and ask in plain language: "What notice period did we agree with this supplier?" It answers and shows the file and page it took the answer from.
The technique is called retrieval-augmented generation. Your files are split into chunks and indexed; when you ask, the system finds the right chunks and hands only those to the model. It doesn't "learn" your data, so you can remove a document and it is gone from the answers the next minute.
What a local LLM needs
Honest numbers, because this is where expectations go wrong:
- A 7–14 billion parameter model runs well on a server with a single modern GPU and 32–64 GB of RAM. That is enough for summaries, drafting and document search for a small team.
- Bigger models give better reasoning but need far more memory. Quantised versions trade a little quality for a lot less hardware.
- Without a GPU it still works, just slowly. Fine for overnight jobs, painful for chat.
My rule: start with the self-hosted, API-model setup, measure what people actually ask, and only buy hardware for a fully local model when the data truly can't leave.
Protecting client data
A self-hosted setup is not automatically safe. The things I always add:
- Redaction before anything leaves: names, phone numbers, account and ID numbers are masked before a question goes to an outside model.
- Permissions that match your folders: if someone can't open the HR folder, the assistant won't quote it to them either.
- An audit log: who asked what, and which documents were used in the answer.
- Backups and updates like any other business system. It is software, it needs care.
When I'd tell you not to bother
If your team only uses AI for general writing and research, with no client data involved, a business plan from a major provider is simpler and perfectly reasonable. Private AI earns its keep when the documents are sensitive, when you need answers with sources, or when per-seat prices add up across a team.
If you are weighing it up, I'm happy to look at what your team actually does and tell you honestly which side you are on. That's the kind of work I do through AI agents and private ChatGPT setups.