Reading collection

Self-Hosted AI: Local Models, Platforms and Agents

Decide when to use a local model runner, a shared inference server, or a self-hosted AI assistant. Trial quality, privacy boundaries, and cost before buying hardware.

Editorial update

Decision in brief

Start with a local model runner such as Ollama for one person and a bounded task with clear examples. Evaluate a shared inference server such as vLLM when several users need one endpoint and someone can own the service. A self-hosted AI assistant or agent adds retrieval, tool access, queues, and permissions. Add those features only once you know the model can reliably handle the underlying job.

Start with a model: One user — Local runner. Shared use — Inference server. Agent features — Add after testing.
Validate the model on a bounded workload before adding retrieval, tools or permissions.

Choose what you need to host

A self-hosted AI model serves prompts and outputs. A self-hosted AI platform also handles applications and serving across users, observability, access controls, and upgrades. An agent adds tools and actions. Drawing the full data path from client to inference, embeddings, and logs makes it clear where user data goes and who can access it.

Ollama documents a local-only mode that disables its cloud models and web search. Verify the rest of the stack independently, including the client, embedding model, connectors, crash logs, and backups. For a shared service, vLLM exposes an OpenAI-compatible HTTP interface. Measure the whole application at the expected concurrency because weights, context cache, and queued requests all contribute to resource demand.

RequirementPilot architectureDecision gate
Private drafting or classification for one userLocal runner, one pinned model, local interfaceExpected task quality and time to first token on hardware you already own.
Shared inference for a teamAuthenticated internal endpoint with a serving engine and defined ownershipLatency and queue behavior at peak load, documented upgrades, and recovery after restart.
Assistant that answers from internal documentsModel plus local embeddings, separate indexes, and a read-only UISource attribution accuracy and realistic tests of document access boundaries.
Self-hosted AI agentBounded tools and review for write actionsTool calls stay within the authorized scope and proposed changes can be inspected or reversed.

What two Hacker News setups exposed

In the M4 Pro Mac Mini discussion, jumploops said prefill latency was the main issue even on an M4 Max with 128 GB of RAM. That experience motivates measuring time to first token on realistic document lengths. A short demo can feel quick while the long-input workflow still misses its latency target.

In a separate discussion on local model quality, anotherCodder described a lower-quality Qwen setup after a chat-template mismatch and warned that sampling defaults can also confound comparisons. When the local answer is incoherent, record the exact model revision, quantization, prompt template, and sampling options. Change one variable at a time before upgrading hardware.

Measure twenty examples before buying hardware

Start with a read-only trial using representative synthetic or approved inputs. Budget actual model serving, storage, electricity, and operator time. The useful comparison is fully loaded monthly cost divided by accepted outputs at the concurrency and quality users need.

  1. Write twenty examples with expected outcomes covering short, typical, and long inputs. For a document assistant, include source answers, an unanswerable example, and a file an ordinary test user should not be able to read.
  2. Pin the runner, weights, quantization, and template. Set a task quality gate and a p95 latency ceiling in advance. Test one cold request, repeated warm requests, and the real peak level of parallel requests.
  3. Record time to first token, total duration, peak memory, failed requests, and whether summaries and citations pass spot checks. Confirm the expected network destinations during the run and test what happens when the endpoint restarts.
  4. Keep the smallest verified architecture that passes the quality, data access, and latency gates. If the best local candidate misses them, compare an approved hosted endpoint on the same twenty inputs before ordering equipment.

Related articles