Home/Solutions/Inference
Today you run vLLM on your own pod — full control, per-second billing. A scale-to-zero serverless endpoint arrives in Phase 2; it does not exist yet and we do not describe it as if it did.
You pay per second and stop whenever you want. No commitment.
Your work probably looks like this
When you put a model API into production, the problem is not running the model; it is that the card keeps burning money at 03:00 when nobody is sending requests.
On an endpoint that is busy by day and empty at night, you pay for waiting for most of the month. A permanently open pod bills its hours regardless of usage.
Shut it down and you save money, but the first user waits for the model to load. That is why most teams never shut the endpoint down — and pay extra.
If every provider forces its own SDK on you, the cost of moving turns into lock-in. On an OpenAI-compatible endpoint the only line that changes should be base_url.
Two paths, an honest comparison
The left column works today. The right one sits in our technical plan as Phase 2 (months 4–8): a vLLM-based, OpenAI-compatible, scale-to-zero endpoint. We are writing down its targets as targets, not as commitments.
| Item | vLLM on your own pod — today | Serverless — Phase 2, soon |
|---|---|---|
| Available today | Yes — live in Phase 0 | No — part of Phase 2 |
| What is billed | Every second the pod is open | Target: only the seconds a request is processed |
| Idle traffic hours | You pay (unless you shut the pod down) | Target: scales to zero, you pay nothing |
| Cold start | None — the process is already up | Target: under 2 seconds |
| Control | Image, driver, vLLM flags and batching are entirely yours | Through a template; less room for tuning |
| API surface | The OpenAI-compatible endpoint you set up | Target: a ready OpenAI-compatible URL |
| Scaling | You raise the pod count yourself | Target: automatic, by request load |
Every row in the right column is a target, not a feature. The done criterion for Phase 2 is written in our technical plan like this: a user pastes a Hugging Face model name and has a production endpoint in 10 minutes. Until that day, this column stays "soon".
The flow
Weights + KV cache + concurrent request load all have to fit in VRAM. An 8B model is comfortable in 24 GB; for 70B you need the 80 GB class or a quantization decision.
If the model files sit on the volume, you do not re-download them when you rebuild the pod. The same volume is reused across several experiments.
Your own image or a ready PyTorch image; you expose the port through the pod connection. On the client side the only thing that changes is base_url.
A permanently open endpoint is the most predictable line on a monthly invoice — but it is still protected by a cap. Notifications at 80% and 95%, pause at 100%.
Seconds that fail the health probe never enter the bill and stay marked on the metrics chart. You do not pay for the time your endpoint was down.
Until serverless ships, this is the most honest way to save: shut it down at night, start it in the morning. Startup targets under two minutes, and you only pay for the seconds you stayed open.
Fictional scenario
Not a real customer: an illustrative example constructed to show the product. The figures are examples.
Cost
The two calculations below are examples and illustrative: the same endpoint, the same card, two different billing shapes. The serverless row in the right column does not exist today; it is a projection built on the Phase 2 target.
| Example scenario (30-day month) | Calculation | Example amount |
|---|---|---|
| Permanently open pod — today | 1× L40S · 730 hr · $0.79/hr | $576.70 |
| Weights volume (200 GB) | 200 GB · 1 month · $0.10/GB-month (example rate) | $20.00 |
| Egress traffic | 300 GB · $0.01/GB (example rate) | $3.00 |
| Today's total (example) | compute + storage + traffic | $599.70 |
| Serverless projection — Phase 2 | 60 GPU-hours of real processing per month · $0.79/hr | $47.40 |
| Serverless — idle hours | 670 hr idle · scale-to-zero target | $0.00 |
These figures are examples and illustrative. The serverless rows are a projection built on today's hourly price; the Phase 2 tariff has not been set. Your real saving depends on how spiky your traffic is: on an endpoint that is busy around the clock, the gap between the two models narrows.
Choosing a GPU
In inference, VRAM carries two things at once: the weights and the KV cache of concurrent requests. What follows is a rough direction; quantization and context length change the table.
| GPU | VRAM | Work it handles comfortably (example) | Example starting price |
|---|---|---|---|
| RTX 3090 | 24 GB | 7B quantized, low concurrency, internal tools | $0.21/hr |
| RTX 4090 | 24 GB | 7B–8B bf16, medium concurrency, fast responses | $0.42/hr |
| L40S | 48 GB | 13B bf16, long context, high concurrency | $0.79/hr |
| A100 PCIe | 80 GB | 70B quantized, multi-user production | $1.32/hr |
| H100 SXM | 80 GB | 70B production load, low-latency target | $2.19/hr |
These are example prices and illustrative; they move with supply and demand in the marketplace. The full list, along with the storage and traffic line items, is on the Pricing page.
Offer search, scorecard score, region pinning: GPU Pods.
What we are targeting in Phase 2, and what does not exist today: Serverless Inference.
FAQ
It sits in our technical plan as Phase 2 and does not exist today. Instead of giving a date we write the done criterion: it ships when a user can paste a model name and have a production endpoint in 10 minutes. Until that day it appears with a "soon" badge in the nav and on this page.
Not when the cause is the infrastructure: seconds that fail the health probe never enter the bill and are marked on the metrics chart. If your own server crashes through its own fault, the GPU is up and running and the charge applies — we are not hiding that distinction.
Yes. When you run vLLM's OpenAI-compatible server on your own pod, the only thing that changes is base_url. The same compatibility is the target for serverless: we do not turn the cost of moving into a lock-in tool.
You can pin your pod to a country; a placement request that would violate the pin is rejected by the API. An EU legal entity, the audited Verified tier and the SOC 2 Type I process are Phase 3 matters — SOC 2 has not been obtained, it is in progress. Details: Sovereignty.
Not today. You set the pod count yourself; you can write your own scaling logic over the API. Automatic scaling by request load is a target belonging to the Phase 2 serverless product.
Get started
You don't pay for seconds that don't work, a hard budget cap, per-second billing. We also tell you whose card is spinning underneath — a failover is an incident record, not silence.
The console runs on the local development address: localhost:3000