Why is self-hosting often more expensive than an API?

GPU instances bill by the hour whether or not you serve traffic. At low or spiky utilization you pay for idle hardware, which is why managed APIs that bill per token usually win until you reach high, steady volume. This tool shows the idle-time waste directly.

How much GPU memory does a model need?

A rough rule is two bytes per parameter at fp16, so a 70B model needs about 140GB just for weights, plus roughly 20% more for the KV cache and activations. Quantizing to int8 or int4 roughly halves or quarters that, letting models fit on smaller GPUs.

What is operational overhead?

Self-hosting is not just GPU rental. You need on-call coverage, security patching, autoscaling, monitoring and incident response. The overhead percentage approximates that engineering cost as a fraction of the raw hardware spend.

When does self-hosting actually pay off?

When utilization is high and steady. If a single GPU is busy most of the day serving real traffic, the per-request cost can undercut API pricing. The break-even depends heavily on your volume, the model size and how much idle time you can eliminate with batching.

Is my data sent anywhere?

No. The calculator runs entirely in your browser. Nothing you enter is uploaded, stored or logged.

Open-Source Model Hosting Cost Calculator

Open-source model hosting cost calculator

“It’s free and open source” only covers the weights — running them is not free. Self-hosting Llama 3, Mistral or Qwen means renting GPUs by the hour, paying for networking and storage, and staffing the operational work to keep it up. This calculator gives you the all-in monthly cost and an honest effective cost per request, so you can compare self-hosting against a managed API on real numbers.

How it works

The model first has to fit in GPU memory. A practical estimate is two bytes per parameter at fp16, plus about 20% for the KV cache and activations:

memory_needed ≈ params_billion × 2 GB × 1.2

The instance is billed for all 730 hours in a month whether busy or idle. The tool adds a networking and storage allowance plus an operational overhead percentage, then divides the total by your monthly request volume to get the effective cost per thousand requests. It also flags how much spend is wasted running below full utilization.

Tips and notes

The single biggest cost lever is utilization. A GPU idling at 40% busy is still 100% billed, so the idle-time waste line is where self-hosting quietly loses money. Push utilization up with continuous batching (vLLM, TGI) and by consolidating workloads onto fewer GPUs. The second lever is quantization: int8 or int4 weights roughly halve or quarter the memory footprint, letting a 70B model run on a smaller, cheaper instance. Self-hosting tends to win only at high, steady volume — for spiky or low traffic a per-token API is usually cheaper and far less operational burden. Run this against the LLM API cost calculator with your real volume to find the break-even.