Decision in brief
Start with a local model runner such as Ollama for one person and a bounded task with clear examples. Evaluate a shared inference server such as vLLM when several users need one endpoint and someone can own the service. A self-hosted AI assistant or agent adds retrieval, tool access, queues, and permissions. Add those features only once you know the model can reliably handle the underlying job.

Choose what you need to host
A self-hosted AI model serves prompts and outputs. A self-hosted AI platform also handles applications and serving across users, observability, access controls, and upgrades. An agent adds tools and actions. Drawing the full data path from client to inference, embeddings, and logs makes it clear where user data goes and who can access it.
Ollama documents a local-only mode that disables its cloud models and web search. Verify the rest of the stack independently, including the client, embedding model, connectors, crash logs, and backups. For a shared service, vLLM exposes an OpenAI-compatible HTTP interface. Measure the whole application at the expected concurrency because weights, context cache, and queued requests all contribute to resource demand.
| Requirement | Pilot architecture | Decision gate |
|---|---|---|
| Private drafting or classification for one user | Local runner, one pinned model, local interface | Expected task quality and time to first token on hardware you already own. |
| Shared inference for a team | Authenticated internal endpoint with a serving engine and defined ownership | Latency and queue behavior at peak load, documented upgrades, and recovery after restart. |
| Assistant that answers from internal documents | Model plus local embeddings, separate indexes, and a read-only UI | Source attribution accuracy and realistic tests of document access boundaries. |
| Self-hosted AI agent | Bounded tools and review for write actions | Tool calls stay within the authorized scope and proposed changes can be inspected or reversed. |
What two Hacker News setups exposed
In the M4 Pro Mac Mini discussion, jumploops said prefill latency was the main issue even on an M4 Max with 128 GB of RAM. That experience motivates measuring time to first token on realistic document lengths. A short demo can feel quick while the long-input workflow still misses its latency target.
In a separate discussion on local model quality, anotherCodder described a lower-quality Qwen setup after a chat-template mismatch and warned that sampling defaults can also confound comparisons. When the local answer is incoherent, record the exact model revision, quantization, prompt template, and sampling options. Change one variable at a time before upgrading hardware.
Measure twenty examples before buying hardware
Start with a read-only trial using representative synthetic or approved inputs. Budget actual model serving, storage, electricity, and operator time. The useful comparison is fully loaded monthly cost divided by accepted outputs at the concurrency and quality users need.
- Write twenty examples with expected outcomes covering short, typical, and long inputs. For a document assistant, include source answers, an unanswerable example, and a file an ordinary test user should not be able to read.
- Pin the runner, weights, quantization, and template. Set a task quality gate and a p95 latency ceiling in advance. Test one cold request, repeated warm requests, and the real peak level of parallel requests.
- Record time to first token, total duration, peak memory, failed requests, and whether summaries and citations pass spot checks. Confirm the expected network destinations during the run and test what happens when the endpoint restarts.
- Keep the smallest verified architecture that passes the quality, data access, and latency gates. If the best local candidate misses them, compare an approved hosted endpoint on the same twenty inputs before ordering equipment.